Transaction Data Latency Reduced from 12 Hours to Under 90 Seconds with Snowflake
Transaction Data Streaming Near Real-Time | Fraud Detection with Data | PCI-DSS Alignment
At a Glance
| Industry | Financial Services |
| Services | Snowflake Implementation, Real-Time Data Streaming, Data Engineering, Security Configuration |
| Challenge | Transaction data took up to 12 hours to become available, limiting its usefulness for time-sensitive fraud detection. |
| Solution | Implemented a Kafka-to-Snowpipe streaming architecture with Snowpark feature engineering pipelines, RBAC, and column-level masking. |
| Key Result | Reduced transaction data latency from 12 hours to less than 90 seconds and went live in 6 weeks. |
About the Client
The client is a financial services company that uses transaction data to help detect fraud.
The company's existing data architecture caused a big delay, so transaction data were almost not in sync with the latest events, and the client's downstream fraud detection activities were also restricted to only very fresh data.
The Challenge
Transaction data took approximately 12 hours to become available for downstream use.
For our fraud detection case, such a time gap greatly limited the data value because we had to react to the fraud with the help of only a few hours' worth of data.
The key challenges were:
- Data Latency of 12-hours: Transaction data was not made available to near-real-time use cases at a sufficiently quick pace.
- Mandate of Fraud Detection: Fraud detection processes needed fresh transaction data a lot.
- Streaming of Data: The company needed a trustworthy architecture to keep transaction events flowing to Snowflake continuously.
- Need for Feature Engineering: Transaction data from the stream was to be prepared to meet the requirements of downstream fraud detection.
- Sensitive Financial Data: It was mandatory to have limited access to transaction data and masking of sensitive fields.
- Safeguarding PCI-DSS Requirements: Security mechanisms had to be activated by putting controls in place before the deployment in the production environment.
The company was looking at re-inventing its transaction data pipeline and at the same time maintaining its high standards regarding safety and production compliance.
The Solution
A real-time streaming architecture close to it was built by Aegis Softtech using Apache Kafka, Snowpipe, and Snowflake.
Through this solution, transaction data were fed in almost nonstop to Snowflake while being processed as ready data for fraud detection processes and security controls were being enforced before the product went live.
Kafka-to-Snowpipe Streaming Architecture
Snowpipe took care of data input in Snowflake continuously as part of the transaction event streaming using Apache Kafka that handled the rest of the stream pipeline.
That was the end of the time-based data moving that was the norm, and in a day and a half, the previous delay was replaced by this pipe, making it possible to have the data at hand so much quicker.
Transaction Data Ingestion Nearly Real-Time
The data pipeline was designed in a way that the flow of transaction data was continuous rather than being held until very long batches were processed.
This significantly cut down the time it took for a transaction to happen and yet for the details of the same transaction and other connected transactions to become available in the Snowflake database.
Snowpark Feature Engineering
Snowpark pipelines were introduced for carrying out feature engineering on the transaction data.
It was then made possible for incoming information to be transformed and prepared within the Snowflake environment for the use cases downstream of fraud detection.
Role-Based Access Control
Role-based Access Control (RBAC) was set up to limit access to transaction data according to the roles of users and systems.
With this, sensitive financial information was given controlled access before the platform went live.
Column-Level Data Masking
Field-level data masking techniques were used for sensitive data fields to be safeguarded.
This provided a further level of protection for transaction data and allowed authorized users and processes to work with the data they needed.
PCI-DSS-Aligned Security Configuration
Security controls around PCI-DSS requirements were set up by the client before the deployment of production.
RBAC and column-level masking were part of the security model to protect sensitive transaction information within the Snowflake environment.
The Production Deployment Only Took 6 Weeks
The whole streaming solution became live in just six weeks, which also covers feature engineering, access controls, data masking, and production deployment.
What Was Achieved?
Snowflake streaming implementation changed how fast the fraud detection departments are fed with transaction data in case of fraud occurrences.
- 12 Hours to Under 90 Seconds: The transaction data arrival time reduced from almost 12 hours to less than 90 seconds.
- 6-Week Production Delivery: The near real-time streaming architecture was implemented and moved into production within 6 weeks.
- Fresher Data for Fraud Detection: The processing of fraud detection had access to transaction information much earlier than before.
- Continuous Data Ingestion: A data pipeline was implemented after replacing the previous slow approach.
- Sensitive Data Protection: Column-level masking helped protect sensitive transaction fields within Snowflake.
- Production-Ready Security: Required security controls were configured before the streaming architecture went live.
Technology Stack
- Snowpipe
- Snowpark
- Snowflake
- Apache Kafka
- Column-Level Masking
- Streaming Data Pipelines
- Feature Engineering
- RBAC
- Securing data against PCI-DSS
How About Almost Real-Time Data in Snowflake?
With Aegis Softtech's Snowflake consulting and implementation expertise, your company can build efficient Snowflake streaming and data engineering solutions. These will allow you to get a fast flow of the massive data from operational systems into the analytics environment.
We can handle everything, starting from Kafka and Snowpipe data ingestion to Snowpark data processing, setting up access controls, and finally deploying the pipeline for production. We'll provide safe pipelines that match your business requirements on tight deadlines.
In order to learn more about how we can assist you or to ask any questions you may have.
Get in touch with any of our data engineering specialists today for the details on Snowflake!
*Client identity is confidential.*