Have built a fintech startup here in Germany ten years ago. The article mentions lots of important things. Here‘s another one:
Time synchronization. It is incredibly important in fintech applications for a number of reasons:
1. *Transaction Ordering:* Financial transactions often need to be processed in the order they were initiated. This is especially crucial in high-frequency trading where trades are often made in milliseconds or microseconds. A small difference in timing could potentially lead to substantial financial gains or losses. Therefore, accurate time synchronization ensures fairness and order in the execution of these transactions.
2. *Security:* Accurate timekeeping helps in maintaining security. For instance, time-based one-time passwords (TOTPs) are widely used in two-factor authentication systems. These passwords are valid only for a short period of time and rely on synchronized clocks on the server and client side.
3. *Audit Trails and Dispute Resolution:* Timestamping transactions can help create a precise audit trail, which is critical for detecting and investigating fraudulent activities. In case of any dispute, a detailed and accurate transaction history backed by synchronized time can help resolve the issue.
4. *Distributed Systems:* In distributed systems, time synchronization is important to ensure data consistency. Many financial systems are distributed over different geographical locations, and transactions need to be coordinated between these systems in an orderly fashion. This requires all servers to have their clocks synchronized.
I am sure there are even more fields where this is relevant.
This was also a surprise to me when we started TigerBeetle.
How clock synchronization protocols can experience unaligned network partitions, where the ledger database cluster is able to continue running, but now with the risk of unsynchronized clocks and far-future timestamps, which can in turn then lead to money being locked up in 2PC payment protocols.
We therefore spent considerable effort [0] on clock synchronization in TigerBeetle, not for the consensus protocol—we never risk stale reads or take a chance with clock error bounds—but rather simply for accurate audit trails and to keep inflight liquidity from being locked up if transactions take too long to get rolled back.
You don't represent money as (<currency>, <decimal amount>), you may want to store it as (<currency>, <decimal amount>, <timestamp>) to be able to apply the correct exchange rate post-facto, and not have to deduce from the transaction history.
It also helps with cross-checks/consistency verification.
Also, preserve the original timezone in the timestamp. It can save many headaches down the road.
> Financial transactions often need to be processed in the order they were initiated.
And then the banks send you info in batches and out of order :) This happened to us more than once. So the team responsible wouldn't settle/cancel a payment for X even if bank said so. They would read a few other sync batches yo make sure that nothing else changed for X.
Because "Financial transactions need to be processed in the order they were initiated" is a must :)
— If you’re using JSON to pass around monetary quantities (eg. from the frontend to the backend), put them in strings as opposed to the native number type. You never know what the serializers and deserializers across languages will do to your numbers (round them, truncate them etc.).
— Start recording the currency of the transactions as early as possible. It can be a separate column in your table.
> Use ISO8601 date and time with offset info, always.
I think that's just sufficient for all use cases everywhere. I've been a software engineer for almost 25 years and, of all the universal truths I've encountered, implicit and unintended changes to datetime offset are the ones I've seen at every single job.
"— If you’re using JSON to pass around monetary quantities (eg. from the frontend to the backend), put them in strings as opposed to the native number type. You never know what the serializers and deserializers across languages will do to your numbers (round them, truncate them etc.)."
I'd go a step further and prefix the strings with an ISO currency code ... to stop someone from just feeding it into their languages int to float converter and assuming that's ok. Only custom built (hopefully safe) converters will work.
Ergh, I get what you're trying to prevent, but this actively breaks those custom safe parsers we build, and now you have to do some additional active parsing. Please don't do this, just set contractual expectations in your API.
I'm generally not a fan of overloading fields, but we took the opportunity to add the currency code when we decided to encode monetary quantities in JSON as strings, e.g. "0.02USD". This has worked well, particularly because we use a money handling library that parses it unchanged.
Don't be afraid to use formal methods. Queues, retries, event sourcing, payment state handling -- there are "global" properties we desire from these systems: certain things we want to make sure are always true and others that we want to make sure eventually happen. For the single process case, which nearly every developer thinks about, it can seem like a solved problem but we live in a concurrent world with network failures and vendors with errors in their own systems: it's nearly impossible for you to think really hard in your head and be sure that your queue retry strategy will maintain those properties that are important to your business. It's nearly impossible to do this with lightweight, informal testing strategies employed by busy software teams.
By modelling your systems you will learn what the important failure modes are and you will get better at designing systems that are resilient and efficient.
Card payment systems are fairly unreliable peer-to-peer messaging systems. Be prepared for a lot of complexity. Using an event-sourcing architecture is really useful here for that "auditing" requirement and for debugging transaction state when the network sends you messages in error, out of order, or they forget to retry themselves when they promised to, when merchants send bad data, when POS systems do weird things, etc.
Various platforms use: UTC, the user’s timezone as set in their dashboard, the user’s detected timezone, the timezone of the server that is generating the reports.
Aggregating or reconciling data from different platforms can be a pain if the timezones aren’t clearly indicated.
I’ve had bank statements that didn’t agree to CSV exports of the same data because the servers generating the two reports were in different timezones.
Haha yeah for the past tax year I I looked at my transaction history and it was missing some that I expected to be there. Turns out, the exchange (Gemini, for crypto) uses UTC, but the IRS docs always tell you that a cutoff is relative to your own time zone.
The transactions were in the late evening on New Year's Eve, central US time, but that was already the next year by UTC, so any readout of "transactions for 2022" would not include them.
That’d be a great subplot for a movie. The antagonist sets up an elaborate money laundering operation that leverages conflicting definitions of the taxable year and moves off shore profits that are “lost” between the systems. Could call it “Black Ink to the Future.
Furthermore, even for “date fields”, consider using a datetime. Every date implicitly exists in a timezone, and if you ignore that ambiguity you’ll get bitten later.
For example, an invoice due on Friday is probably actually due by close of business (5pm say) in the timezone your business operates, and if created at 11pm it would be processed the next day (or even on Monday, don’t get me started about business day calculations).
> Every date implicitly exists in a timezone, and if you ignore that ambiguity you’ll get bitten later.
Solid advice, and must come from painful burns. I've been preaching from the same book for a few years now: a timestamp without timezone offset is worse than useless.
Or as the DB expert in previous job so eloquently put it... Timestamp with zone tells you when an event actually happened. A timestamp without zone or offset is equal to wall clock time inside a windowless room, which itself is in an unspecified location somewhere on the planet.
We stored and transmitted exclusively UTC, which put the burden on the UI at display time (trivial) and backend business logic (complex, but simplified with robust utility functions that everyone understood well) in accounting for timezones when material.
We only had a dozen issues due to it so I believe it worked pretty well compared to the war stories I heard from comparable companies in the fintech space.
That's why Datetime data should be handled in a type that includes this data (e.g. DateTimeOffset (1) ) and exchanged as ISO 8601 with offset. e.g. "2023‐06‐28T15:55:22+01:00"
#2 and #3 are pretty solid advice. But the rest of them are nothing specific to fintech. I will also offer a few additional points.
1. Never record an amount without its currency.
2. Reconcile all your data all the time. Never let any data go unaccounted for.
3. Maker-checker is a powerful concept. Embrace it to the fullest across your system.
4. You will be dealing with all sorts of non-standardized financial integrations. A lot. Think adapter pattern as early as possible.
5. You will be answering to multiple regulatory agencies. Create boundaries between them within your system and reduce the surface of compliance as much as possible.
I think this article covers many of the main points, but could be worded/structured better. There are terms of art that more precisely refer to the concepts you need.
For “using floating point data types”, it’s even worse; you often need to use strings, for example if you need to store a bank account number “01234567789” will have the leading zero stripped if you use a numeric type.
“Updating transactions” would be better phrased as “use an append-only log / evented architecture”. (Also, “use a double-entry ledger” is probably the most valuable advice I could have sent myself prior to getting into FinTech.)
“Be careful with retry” should be more strictly “use idempotent operations” and link to the canonical Stripe article on idempotency keys (https://stripe.com/blog/idempotency).
Another important one to think about is bitemporality. “Created at” vs “effective at”. Not obvious at first and you’ll have some painful migrations if you don’t build it in. Fowler has a good overview here: https://martinfowler.com/eaaDev/timeNarrative.html.
Edit to add - the advice that maybe using a NoSQL database is pretty bad IMO. I’d advise in the opposite direction - use SERIALIZABLE isolation in a SQL database. Read up on your Aphyr blog posts before trying to do anything distributed. Be paranoid about race conditions / serialization anomalies. If you eventually hit performance issues you need to think hard about what anomalies your access patterns might be subject to. (Obviously HFT won’t use serializable SQL).
> Over the last decade, fintechs using RDBMS databases to record transactions have built features onto their databases like a journal of all changes to all data and clever ways of making sure the data hasn’t been modified by storing checksums of data as transactions are added. (under “Updating transactions”)
It is worth noting that many of these methods don't prevent tampering, they only make it visible when you look for it at which point the fact you are explicitly looking for it (rather than having been alerted to a potential issue) might imply it is too late.
The recommendation that "developers should use integers to represent money" (with an implicit 2 decimal places, that will work for many but not all currencies) is not a great one.
If your language has a dedicated type for monetary amounts, use that. (1)
If it does not, but you can make a value object to represent, e.g. amount and currency code, then do that.
If however, your language does not have a dedicated type for monetary amounts, or one cannot be trivially built or retrieved as a package (2), then you should ask yourself if it is really a suitable language for financial tasks.
a) On web api, so this is by definition data interchange, not "in a language".
b) with a currency code to cross check it.
c) implicitly "in cents" which needs further documentation - does this mean "pence" when the currency is GBP? How does this work for for JPY? BHD?
d) cannot represent fractions of a cent or penny.
So: this int plus currency code plus docs is OK but not great, a lowest common denominator format for interchange, requires further documentation and cross-checking as "100 USD" does not mean 100 bucks and the conversion is currency-specific, and cannot represent all values.
I wouldn't refuse to convert _to and from_ this format at the edges of my application, for data interchange, but the conversion to something clearer and richer IMHO should remain there.
Payment APIs are far from the only use case in fintech, for interest calculations you do have to care about fractions of a cent or penny.
In fact, if your case is "I call the stripe api" ... are you sure you're a fintech and not an online store? Get back to me when you have to interop with FiServ, MasterCard or SAP.
It's nice if your language has support for monetary amounts, but usually you end up using multiple languages that interact with the same database model, and you still end up using a built in numeric datatype in your RDBMS of choice.
Three additional 'mistakes' to prevent when dealing with money representations:
1. The definition of a currency might change. For example, some years ago Iceland decided to change the exponent of ISK from 2 to 0. Currencies have different versions.
2. As a FinTech you probably have integrations with many third parties, they don't change their exponents for a currency at the same time. Keep track of what third parties think the correct exponent is at any point in time, and convert between your representation and their representation. Otherwise, you'll have interesting incidents (e.g. transferring 100x the intended amount of ISK).
3. At first you think that counting minor units as an integer is enough, and then you need to start accounting for fractions of cents because sales people sold something for a fee of $0.0042 per transaction. If your code rounds all these fees to $0.00 you don't make any money.
1, 2 - Yeah, there's always a sanity checking and conversion layer around the 3rd party. Currencies do indeed sometimes have 2 versions in play there.
3 - Indeed, the currency type that I referenced "is appropriate for financial calculations that require large numbers of significant integral and fractional digits and no round-off errors"
Readit News
Time synchronization. It is incredibly important in fintech applications for a number of reasons:
1. *Transaction Ordering:* Financial transactions often need to be processed in the order they were initiated. This is especially crucial in high-frequency trading where trades are often made in milliseconds or microseconds. A small difference in timing could potentially lead to substantial financial gains or losses. Therefore, accurate time synchronization ensures fairness and order in the execution of these transactions.
2. *Security:* Accurate timekeeping helps in maintaining security. For instance, time-based one-time passwords (TOTPs) are widely used in two-factor authentication systems. These passwords are valid only for a short period of time and rely on synchronized clocks on the server and client side.
3. *Audit Trails and Dispute Resolution:* Timestamping transactions can help create a precise audit trail, which is critical for detecting and investigating fraudulent activities. In case of any dispute, a detailed and accurate transaction history backed by synchronized time can help resolve the issue.
4. *Distributed Systems:* In distributed systems, time synchronization is important to ensure data consistency. Many financial systems are distributed over different geographical locations, and transactions need to be coordinated between these systems in an orderly fashion. This requires all servers to have their clocks synchronized.
I am sure there are even more fields where this is relevant.
How clock synchronization protocols can experience unaligned network partitions, where the ledger database cluster is able to continue running, but now with the risk of unsynchronized clocks and far-future timestamps, which can in turn then lead to money being locked up in 2PC payment protocols.
We therefore spent considerable effort [0] on clock synchronization in TigerBeetle, not for the consensus protocol—we never risk stale reads or take a chance with clock error bounds—but rather simply for accurate audit trails and to keep inflight liquidity from being locked up if transactions take too long to get rolled back.
[0] https://tigerbeetle.com/blog/three-clocks-are-better-than-on...
You don't represent money as (<currency>, <decimal amount>), you may want to store it as (<currency>, <decimal amount>, <timestamp>) to be able to apply the correct exchange rate post-facto, and not have to deduce from the transaction history.
It also helps with cross-checks/consistency verification.
Also, preserve the original timezone in the timestamp. It can save many headaches down the road.
And then the banks send you info in batches and out of order :) This happened to us more than once. So the team responsible wouldn't settle/cancel a payment for X even if bank said so. They would read a few other sync batches yo make sure that nothing else changed for X.
Because "Financial transactions need to be processed in the order they were initiated" is a must :)
— If you’re using JSON to pass around monetary quantities (eg. from the frontend to the backend), put them in strings as opposed to the native number type. You never know what the serializers and deserializers across languages will do to your numbers (round them, truncate them etc.).
— Start recording the currency of the transactions as early as possible. It can be a separate column in your table.
— Use TIMESTAMPTZ. Always.
When you’re using JSON to pass around datetime data, Use ISO8601 date and time with offset info, always.
e.g. "transactionDate": "2023‐06‐28T15:55:22.511Z"
I think that's just sufficient for all use cases everywhere. I've been a software engineer for almost 25 years and, of all the universal truths I've encountered, implicit and unintended changes to datetime offset are the ones I've seen at every single job.
I'd go a step further and prefix the strings with an ISO currency code ... to stop someone from just feeding it into their languages int to float converter and assuming that's ok. Only custom built (hopefully safe) converters will work.
But since this is not a human visible field, this is probably irrelevant, right?
By modelling your systems you will learn what the important failure modes are and you will get better at designing systems that are resilient and efficient.
Card payment systems are fairly unreliable peer-to-peer messaging systems. Be prepared for a lot of complexity. Using an event-sourcing architecture is really useful here for that "auditing" requirement and for debugging transaction state when the network sends you messages in error, out of order, or they forget to retry themselves when they promised to, when merchants send bad data, when POS systems do weird things, etc.
Various platforms use: UTC, the user’s timezone as set in their dashboard, the user’s detected timezone, the timezone of the server that is generating the reports.
Aggregating or reconciling data from different platforms can be a pain if the timezones aren’t clearly indicated.
I’ve had bank statements that didn’t agree to CSV exports of the same data because the servers generating the two reports were in different timezones.
The transactions were in the late evening on New Year's Eve, central US time, but that was already the next year by UTC, so any readout of "transactions for 2022" would not include them.
[1] e.g. https://www.irs.gov/taxtopics/tc301
For example, an invoice due on Friday is probably actually due by close of business (5pm say) in the timezone your business operates, and if created at 11pm it would be processed the next day (or even on Monday, don’t get me started about business day calculations).
Solid advice, and must come from painful burns. I've been preaching from the same book for a few years now: a timestamp without timezone offset is worse than useless.
Or as the DB expert in previous job so eloquently put it... Timestamp with zone tells you when an event actually happened. A timestamp without zone or offset is equal to wall clock time inside a windowless room, which itself is in an unspecified location somewhere on the planet.
We only had a dozen issues due to it so I believe it worked pretty well compared to the war stories I heard from comparable companies in the fintech space.
1) https://learn.microsoft.com/en-us/dotnet/api/system.datetime...
1. Never record an amount without its currency.
2. Reconcile all your data all the time. Never let any data go unaccounted for.
3. Maker-checker is a powerful concept. Embrace it to the fullest across your system.
4. You will be dealing with all sorts of non-standardized financial integrations. A lot. Think adapter pattern as early as possible.
5. You will be answering to multiple regulatory agencies. Create boundaries between them within your system and reduce the surface of compliance as much as possible.
For “using floating point data types”, it’s even worse; you often need to use strings, for example if you need to store a bank account number “01234567789” will have the leading zero stripped if you use a numeric type.
“Updating transactions” would be better phrased as “use an append-only log / evented architecture”. (Also, “use a double-entry ledger” is probably the most valuable advice I could have sent myself prior to getting into FinTech.)
“Be careful with retry” should be more strictly “use idempotent operations” and link to the canonical Stripe article on idempotency keys (https://stripe.com/blog/idempotency).
Another important one to think about is bitemporality. “Created at” vs “effective at”. Not obvious at first and you’ll have some painful migrations if you don’t build it in. Fowler has a good overview here: https://martinfowler.com/eaaDev/timeNarrative.html.
Edit to add - the advice that maybe using a NoSQL database is pretty bad IMO. I’d advise in the opposite direction - use SERIALIZABLE isolation in a SQL database. Read up on your Aphyr blog posts before trying to do anything distributed. Be paranoid about race conditions / serialization anomalies. If you eventually hit performance issues you need to think hard about what anomalies your access patterns might be subject to. (Obviously HFT won’t use serializable SQL).
This goes for any "number" that is actually an identifier. Phone numbers aren't really numbers either for example.
Rule of thumb: if it doesn't make sense to do math with it, it should probably be a numeric type unless there's a very good reason.
It is worth noting that many of these methods don't prevent tampering, they only make it visible when you look for it at which point the fact you are explicitly looking for it (rather than having been alerted to a potential issue) might imply it is too late.
The number one thing they crave is immutability. But when that's not an option, tamper-evident comes as a close second.
If your language has a dedicated type for monetary amounts, use that. (1)
If it does not, but you can make a value object to represent, e.g. amount and currency code, then do that.
If however, your language does not have a dedicated type for monetary amounts, or one cannot be trivially built or retrieved as a package (2), then you should ask yourself if it is really a suitable language for financial tasks.
1) https://learn.microsoft.com/en-us/dotnet/api/system.decimal
2) https://github.com/shopspring/decimal https://www.npmjs.com/package/ts-money
Adyen uses integers [2]
Square uses integers [3]
So not sure that the industry would agree with you
[1] https://stripe.com/docs/api/prices/create#create_price-unit_...
[2] https://docs.adyen.com/api-explorer/Checkout/70/post/payment...
[3] https://developer.squareup.com/reference/square/objects/Mone...
a) On web api, so this is by definition data interchange, not "in a language".
b) with a currency code to cross check it.
c) implicitly "in cents" which needs further documentation - does this mean "pence" when the currency is GBP? How does this work for for JPY? BHD?
d) cannot represent fractions of a cent or penny.
So: this int plus currency code plus docs is OK but not great, a lowest common denominator format for interchange, requires further documentation and cross-checking as "100 USD" does not mean 100 bucks and the conversion is currency-specific, and cannot represent all values.
I wouldn't refuse to convert _to and from_ this format at the edges of my application, for data interchange, but the conversion to something clearer and richer IMHO should remain there.
Payment APIs are far from the only use case in fintech, for interest calculations you do have to care about fractions of a cent or penny.
In fact, if your case is "I call the stripe api" ... are you sure you're a fintech and not an online store? Get back to me when you have to interop with FiServ, MasterCard or SAP.
Three additional 'mistakes' to prevent when dealing with money representations:
1. The definition of a currency might change. For example, some years ago Iceland decided to change the exponent of ISK from 2 to 0. Currencies have different versions.
2. As a FinTech you probably have integrations with many third parties, they don't change their exponents for a currency at the same time. Keep track of what third parties think the correct exponent is at any point in time, and convert between your representation and their representation. Otherwise, you'll have interesting incidents (e.g. transferring 100x the intended amount of ISK).
3. At first you think that counting minor units as an integer is enough, and then you need to start accounting for fractions of cents because sales people sold something for a fee of $0.0042 per transaction. If your code rounds all these fees to $0.00 you don't make any money.
3 - Indeed, the currency type that I referenced "is appropriate for financial calculations that require large numbers of significant integral and fractional digits and no round-off errors"