Ever get stuck dealing with mountains of data and sky-high storage bills? That’s exactly what a major US cable operator faced, drowning in a sea of phone calls, chats, and email recordings. Storing all that data with a third-party vendor was becoming a budget black hole.
Here at ⋮IWConnect we love a good data challenge. The cable operator needed a way to:
- Ditch the storage fees: Move all that data in-house for better control and cost savings.
- Automate the data shuffle: No more manual downloads, just set it and forget it.
- Keep it safe and secure: Recordings and data are sensitive, so security was paramount.
The challenge that we faced was which technology to choose to be the most cost-effective and at the same time the best solution for processing the data. After reviewing plenty of different options the choice was for system that would be synergy between Microsoft’s .NET framework and Amazon Web Services.
Solution 1: Automated Data Transfer with Serverless Functions
The application was created and based on three lambda functions. The first lambda function was invoked with API Gateway, which was called from another application scheduled on a daily (or hourly) basis, demonstrating a seamless serverless architecture deployment.
The first lambda function with SQS (queue) asynchronously triggered the second lambda function. In the SQS (which is triggering the second lambda function), a message was sent that contained the processing type (“recordings”—if the purpose is to get and upload the recordings or “metadata”—if the purpose is to get and store the metadata), start date, and end date.
In the second lambda function, based on the SQS message values (if there is tag “recordings” or “metadata”), different functions were called to execute various tasks.
If the tag is “recordings”, then all the Contacts IDs for a particular period were collected. If the tag is “metadata”, we were just passing the message in the new SQS queue that invoked the third lambda function.
Then, from the second lambda function with SQS, the third lambda function will be invoked to execute the process. Again, we check the tag’s value. If it is “recordings,” a different instance of the lambda function will be invoked for every contact ID.
The whole processing is done in the third Lambda function. From there, the APIs for getting the recording for the corresponding contact ID are called.
After the recording is obtained from the API, the file is uploaded to AWS S3, which is configured for efficient data retrieval.
Solution 2: Automated File Deletion from Third-Party Storage
An application was created to remove audio files from third-party storage by calling APIs to delete the files. The application was running scheduled on an AWS EC2 instance every day, but plans are underway to deploy it on Azure for better efficiency. It was getting the file names (with full file paths previously gathered from the application for uploading files), processing them, and calling APIs for file deletion.
Benefits from the application
The application automated collecting the audio files and metadata using serverless applications. Instead of manually downloading the audio files, the application was scheduled to call APIs for that, deploy necessary components, and perform all the processing automatically.
This app offered completely automated processing from start to finish, reducing manual intervention, human error, and processing time. The automation enhances efficiency and allows staff to focus on more strategic tasks. Having all data stored in-house is also beneficial, ensuring greater control over security protocols and compliance measures. This mitigates risks associated with data breaches and ensures adherence to regulatory requirements. With this application, we achieved significant cost savings of more than 30% by eliminating the need for expensive third-party solutions or services. With everything in-house, no recurring subscription fees or additional costs are associated with accessing data. Without relying on external vendors for support and maintenance, eliminate the need for further expenses and streamline your deployment. The application is self-sufficient, reducing downtime and dependency on external parties.
Conclusion
This project demonstrates how a well-designed AWS solution can optimize data management for businesses of all sizes. By leveraging serverless technologies and fostering synergies between AWS and .NET, we were able to deliver cost savings, improve efficiency, and enhance data security for a leading US cable operator.
