RPA for Web Scraping and Streamlining Data Comparison for Enhanced Accuracy at a Leading HR Solutions Provider


A leading HR and talent management solutions provider leveraged Robotic Process Automation (RPA) to streamline their data comparison process. The Client was facing a time-consuming and error-prone challenge of manually extracting and cross-referencing covered individuals’ data from insurance carrier websites with their internal system. This labor-intensive process lacked automation, resulting in potential data inconsistencies, missing individuals, and payment inaccuracies that needed manual detection and resolution.

The Process Prior Implementing RPA

The manual comparison involved pulling data from carrier websites, manual data extraction, and comparing it with SSRS extracts from the internal system. Detected errors were addressed manually, leading to inefficiencies and potential inaccuracies.

Scope and Must-Have

  • Develop a bot for web scraping carrier websites to gather data.
  • Store extracted data locally in the Client’s database.
  • Implement a systematic method to compare data and identify discrepancies.
  • Integrate an alerting system or reporting mechanism for users to address data inconsistencies.
  • Create a user-friendly, non-programmer-accessible method for adding new carriers.

Proposed Solution & Robotic Process Automation

An automated process was devised to identify and resolve data inconsistencies between insurance carrier websites and Client’s internal data. The automated process runs daily and performs the following steps:

  • Initial Data Collection: On the first run, the bot scrapes data from carrier websites for all employees, storing it in the database. Subsequent runs focus on employees with changes made on the previous day.
  • SQL Query Execution: The bot extracts a list of relevant employees from the database by executing an SQL query. This list includes newly added employees, changes in termination dates, plan modifications, and more.
  • Web Scraping and Data Storage: The bot logs into carrier websites and extracts data for each employee, using SSN as a search criterion. Extracted data is stored in a dedicated database with new tables created for storage.
  • Comparison and Reporting: An SSRS report is generated, displaying data from both Employer Flexible and carrier sources. Discrepancies are highlighted, providing clear visibility of inconsistencies.

Benefits and Outcome of Using RPA

By implementing the automated process, the Client achieved a significant improvement in data accuracy and efficiency:

  • Time Savings: Manual efforts reduced significantly, enhancing overall productivity.
  • Data Accuracy: Automated comparison ensures discrepancies are promptly detected and addressed.
  • Enhanced Visibility: The SSRS report offers an accessible overview of data inconsistencies, enabling proactive resolution.
  • Process Optimization: The streamlined process allows for timely corrective actions before payment processes commence.

In conclusion, the Client successfully eliminated manual data comparison bottlenecks, embraced automation, and empowered their team to focus on strategic initiatives, ultimately contributing to a more accurate and efficient data management process.