Share
During the month of April, we made the big announcement of the migration of Máximo's card portfolio, the Peruvian fintech that now uses our Card issuance and processing solution. In this article, we will present technical details of how we carried out this critical processing, some of the tools we use to continue building our entire fintech infrastructure, and how we at Pomelo integrate new clients quickly and scalably. Let's go!
The challenge of card migration
The card migration process and the transition of the associated transaction processing are challenging from different technical perspectives. Before focusing on how we prepare this process at Pomelo, let's briefly understand what this transfer entails:
First, the new card issuer and processor, in this case Pomelo, need to have the card data that was previously held by another issuer/processor, so that when the network (Visa or Mastercard) starts sending transactions to the new processor, it can authorize and process them correctly. This information transfer must be as fast as possible, as transaction processing is interrupted during the switchover, and this time must be minimized to affect cardholders' activity for the shortest time possible.- Secondly, the arrival of new transfers, and consequently, an increase in traffic, tests the scalability of our infrastructure. Our services must be able to handle a new load, new communications, and primarily, they must alert us quickly in case the process does not go as expected, making monitoring a vital role.
- And finally, the transferred information requires secure handling, as it is financial information covered by PCI-DSS. This imposes additional challenges in communications that we must consider due to the criticality and sensitivity of the information.
Keys to the migration
From this analysis, the most important points on which our solution must focus emerge:
- Processing times: part of the performance of our solution to transfer information and avoid downtime;
- The scalability of our issuance and processing infrastructure: both for processing data transfers and supporting the new transaction load;
- Security: particularly in transfer and communications;
- Monitoring: to detect and correct any unforeseen events at any stage of the migration;
Let's look at how we consider these four pillars in our solution point by point!
Transferring card information
As mentioned, the transfer must happen as quickly as possible, so we must focus on performance and avoid downtime. To achieve this, we designed a specific solution, with an architecture like the following:
The card information to be migrated is managed by a component we call Pomelo Card Migrator. This component is responsible for interpreting a file that contains the card information to be migrated, ensuring the correct processing of the entire set of cards, and finally generating an output file with the migration results.
This component will execute calls to other internal services, such as cardholder user and card services, to ensure their integrity and persistence. Additionally, it will run with the aim of generating result files, thus performing mostly I/O work.
How can we be optimal with this behavior?
At Pomelo, we work with a diverse tech stack, part of which is Kotlin, which we leverage to the fullest for this component. Its handling of non-blocking I/O operations, combined with its capability for concurrent operations through coroutines, allows us to have a component with a complex task, such as processing thousands of card records, making remote calls, written in a simple and high-performance manner, utilizing the full processing capacity of our resources. The concept of structured concurrency, along with the reactive handling of flows, are powerful tools that Kotlin provides, which facilitate the use and development of our components.
Using message queues
Each card processed is done through publishing and consuming messages, using AWS SQS. This simple idea of deferring and separating processing gives us key features:
- It allows us to horizontally scale consumption for card migration and persistence in our infrastructure. This substantially improves performance by simply adding a new instance. While this scalability can be automated, we have chosen in this case to maintain manual control, knowing the optimal load for our services;
- It provides us resilience, knowing that processing can be retried in the face of potential errors such as a networking error. Additionally, the incorporation of a Dead Letter Queue allows us to isolate specific erroneous cases, understand them, and quickly resolve them.
To store the results of the migration, the use of another AWS product, DynamoDB, contributed to the solution, considering its versatility, scalability, and ability to support large amounts of writing. We optimized our model by using the Single Table Design technique, making our queries easy to write to generate metrics and results.
All these technologies communicate and collaborate to form a powerful, maintainable, and scalable solution. We delve into this point further below!
The scalability of our infrastructure
Thanks to the use of Kubernetes, combined with KEDA and leveraging AWS's EKS, we were able to confirm that our auto-scaling policies were working correctly. This was one of our first milestones when planning and executing the migration of Máximo.
While we were very confident that the Pomelo Card Migrator scales easily, thanks to its design and also the tools we used to configure it, we wanted to test it and did so with various load tests simulating two types of traffic:
- the migration load, spike tests were appropriate
- and to verify that our services could support the traffic of a new client, traditional load tests.
To find the optimal number of instances for message consumption, balancing costs, performance, and the health of our services, we established various scenarios: having more or fewer consuming instances in Pomelo Card Migrator - even varying our internal concurrency configuration and resources of our instances - combined with different statuses of cardholder and card services.
Conducting these types of tests is a fundamental part of our development, especially considering the importance of the migration process.
The spike caused by the transfer: from 26 RPM to over 400 RPM. Our architecture auto-scales to prevent service degradation.
If you want to know more about this topic, in this blog article we tell you everything!
Securing the process
As Pomelo is PCI-DSS Compliant, it is our duty to maintain information security. We use several technologies to achieve this goal, among which it is worth highlighting:
For file transfer
The use of asymmetric cryptography. For example, file transfers between Pomelo and Máximo - and all our clients - are encrypted using the PGP scheme. The use of this technique ensures that only Pomelo can read cardholder and card information.
For communication between components
Thanks to the use of the Istio tool as a service mesh, we can easily establish authorization policies using the Authorization Policies functionality. It is an easy-to-use tool that provides an effective security layer, which we leverage in our architecture to ensure that communication between our services is secure.
Below is an example of how one of the configurations we apply looks, to understand how we can declare access to an endpoint of our card service declaratively:
name: cards-issuing-service-name namespace: issuing-and-processing-namespace
AuthorizationPolicy:
- name: deny-everything-but-migrator-service
action: DENY
rules:
- from:
- source:
notPrincipals:
- "cluster.local/ns/cluster-name/sa/migrator-service"
to:
- operation:
methods: ["POST"]
paths: ["/v1"]
These two are not the only tools we use, as keeping payment card data secure requires dedication. You can learn more about this in our article on PCI-DSS certification.
About monitoring
Each component of our architecture is monitored in different ways, but our favorite tool, which proved key to the success of the migration, is New Relic.
The quick integration and the ability to build dashboards and alerts, combining information from all components of the solution, help us have clear visibility of the health of the services and processing during the execution of the migration. And not only for the card migration but also for the processing of new transactions, a fundamental aspect of the migration.
In addition to that, the ability to have, for example, SQS metrics along with the status of services and partial migration results in one place significantly simplifies the monitoring task and enhances our ability to react to abnormal metrics and alerts.
For example, a part of our dashboard, focusing on the component we described:
While New Relic provides monitoring tools, it is also useful to combine all the information and metrics it produces with business metrics: understanding which types of cards were migrated, along with the transactions and rejections, gives us a complete visibility of the live migration process. For example, being able to see rejection metrics compared to general rejection metrics alerts us to potential issues when authorizing transactions. We believe that this practice, which combines both data sources, is a key factor in our monitoring effectiveness, and one that we aim to continue in all our functionalities.
Recapping, we have covered the end-to-end implementation of our migration process, emphasizing different essential aspects and delving into technical concepts that not only apply to our migration process but to any component of our infrastructure.
This is just a part of our technologies and practices when it comes to developing, monitoring, and scaling our architecture to be robust and secure at the regional level. We invite you to share and discuss other alternative designs!
If you have issued cards and are thinking about changing providers, we invite you to learn more details about the process in our documentation.
