Tell that to Synapse customers. Many millions of dollars are missing.
Banks have to follow strict rules to account for where all the money goes. But the way fintechs work, they usually just have one or a couple underlying "FBO" accounts where all the pooled money is held, but then the fintech builds a ledger on top of this (and, as the article points out, to varying levels of engineering competence) to track each individual customer's balance within this big pool of money. In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.
FWIW, after seeing "how the sausage is made", I would never put money into a fintech depository account. Use a real bank. Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money.
One of the things it took a little time to wrap my head around when I started working at Google was trading off reliability or correctness for scaling.
I had previously built things like billing systems and small-scale OLTP web applications, where the concept of even asking the question of whether there could be any acceptable data loss or a nonzero error rate hadn't occurred to me. It was eye-opening to see, not so much that if you're doing millions of qps, some are going to fail, but more the difference in engineering attitude. Right this instant, there are probably thousands of people who opened their Gmail and it didn't load right or gave them a 500 error. Nobody is chasing down why that happened, because those users will just hit reload and go on with their day. Or from another perspective, if your storage has an impressive 99.99999% durability over a year, when you have two billion customers, 200 people had a really miserable day.
It was a jarring transition from investigating every error in the logs to getting used to everything being a little bit broken all the time and always considering the cost before trying to do something about it.
This is the type of thing where it helps to hire the right people from the start. You hire a bunch of leetcode experts, but never stop to ask if, besides conjuring data structures and algos in their minds they actually can build the thing you want to build. If your people know how to build the thing, you don't need to sacrifice growth, it gets built right from the start.
Sometimes you need engineers that have some other type of education, maybe accounting, maybe finance, maybe biology. I always thought that the most important part of my career was understanding every industry I built for, deeply, and knowing experts in those areas so you could ask the really important questions. That is problem solving and engineering. The rest is programming/coding.
This reinforces for me the importance of domain knowledge in engineering leadership. i.e. if you work for a finance company you need to have a decent understanding of finance in order to make the correct technical decisions and tradeoffs, same goes for journalism, same goes for commerce etc. etc. Successful organisations I've worked for have always included domain specific, non-technical, questions to tech team interviews whilst some of the most technically acomplished teams I've worked with have floundered through a lack of domain insight.
Pardon me an old story... I never built a double entry accounting system but decades ago I did build a billing system for a internet/telcom startup that grew to a modest 8 figures revenue.
By accident and not knowing any better as a young dev, I ended up building the billing logic from day one, and for better and worse building it in two places in the system (on a consumer-facing billing webpage, and on a separate backed process that generated invoices and charged credit cards.)
It turned out to be remarkably hard to keep them in sync. We were constantly iterating trying to get traction as we burned down our capital, releasing new products and services, new ways of discounting and pricing (per use, per month, first X free, etc), features like masterpayer/subaccounts for corporate accounts, user-assignable cost centers, tax allocation to those cost centers with penny allocation, etc such that new wrinkles and corner cases would keep popping up causing the numbers on my two screens/methods not to match.
Being personally responsible for the billing, I would go over all the invoices by hand for a couple days each month to insure they matched before we charged the credit cards and mailed out printed invoices as a final check to prevent mistakes. There was always/often some new problem I'd find affecting one or a small handful of customers which I would then fix the code before we billed. I never felt good letting go and not doublechecking everything by hand.
I thought about refactoring the billing logic to occur in one place to eliminate these mismatches and my manual crosschecking, but after a lot of thought I realized I wasn't comfortable with a single codebase and liked having two codebases as it helped me catch my own errors. I then just made it easier and easier to run automate and crosschecks between the two. The billing code was a little too gnarly to be proud of, but I was very proud of the outcome in how accurate our billing was, the lack of complaints and many near misses we avoided for many years. I do feel twinges of guilt for the complexity I left my successors but I still don't really regret it.
After that experience, the motivation for double entry bookkeeping has always made a lot of sense to me. I had sort of reinvented it in my own hacky way with double logic billing code to prevent my mistakes from causing problems for my customers...
hn_throwaway_99 ·8 days ago
Banks have to follow strict rules to account for where all the money goes. But the way fintechs work, they usually just have one or a couple underlying "FBO" accounts where all the pooled money is held, but then the fintech builds a ledger on top of this (and, as the article points out, to varying levels of engineering competence) to track each individual customer's balance within this big pool of money. In Synapse's case, their ledger said the total amount of all of their individual customer balances ended up being much more than the actual funds held in the underlying FBO accounts. Lots of folks are assuming fraud but I'm willing to put money that it was just a shitty, buggy ledger.
FWIW, after seeing "how the sausage is made", I would never put money into a fintech depository account. Use a real bank. Fintechs also often put out the fake promise that deposits are FDIC insured, but this only protects you if the underlying bank goes belly up, not if the fintech loses track of your money.
See https://www.forbes.com/sites/zennonkapron/2024/11/08/what-th...
Show replies
masto ·7 days ago
I had previously built things like billing systems and small-scale OLTP web applications, where the concept of even asking the question of whether there could be any acceptable data loss or a nonzero error rate hadn't occurred to me. It was eye-opening to see, not so much that if you're doing millions of qps, some are going to fail, but more the difference in engineering attitude. Right this instant, there are probably thousands of people who opened their Gmail and it didn't load right or gave them a 500 error. Nobody is chasing down why that happened, because those users will just hit reload and go on with their day. Or from another perspective, if your storage has an impressive 99.99999% durability over a year, when you have two billion customers, 200 people had a really miserable day.
It was a jarring transition from investigating every error in the logs to getting used to everything being a little bit broken all the time and always considering the cost before trying to do something about it.
Show replies
figassis ·8 days ago
Sometimes you need engineers that have some other type of education, maybe accounting, maybe finance, maybe biology. I always thought that the most important part of my career was understanding every industry I built for, deeply, and knowing experts in those areas so you could ask the really important questions. That is problem solving and engineering. The rest is programming/coding.
Show replies
tomgp ·8 days ago
Show replies
gregw2 ·8 days ago
By accident and not knowing any better as a young dev, I ended up building the billing logic from day one, and for better and worse building it in two places in the system (on a consumer-facing billing webpage, and on a separate backed process that generated invoices and charged credit cards.)
It turned out to be remarkably hard to keep them in sync. We were constantly iterating trying to get traction as we burned down our capital, releasing new products and services, new ways of discounting and pricing (per use, per month, first X free, etc), features like masterpayer/subaccounts for corporate accounts, user-assignable cost centers, tax allocation to those cost centers with penny allocation, etc such that new wrinkles and corner cases would keep popping up causing the numbers on my two screens/methods not to match.
Being personally responsible for the billing, I would go over all the invoices by hand for a couple days each month to insure they matched before we charged the credit cards and mailed out printed invoices as a final check to prevent mistakes. There was always/often some new problem I'd find affecting one or a small handful of customers which I would then fix the code before we billed. I never felt good letting go and not doublechecking everything by hand.
I thought about refactoring the billing logic to occur in one place to eliminate these mismatches and my manual crosschecking, but after a lot of thought I realized I wasn't comfortable with a single codebase and liked having two codebases as it helped me catch my own errors. I then just made it easier and easier to run automate and crosschecks between the two. The billing code was a little too gnarly to be proud of, but I was very proud of the outcome in how accurate our billing was, the lack of complaints and many near misses we avoided for many years. I do feel twinges of guilt for the complexity I left my successors but I still don't really regret it.
After that experience, the motivation for double entry bookkeeping has always made a lot of sense to me. I had sort of reinvented it in my own hacky way with double logic billing code to prevent my mistakes from causing problems for my customers...
Show replies