Where to learn how to build an A/B testing tool?
14 points ·
armank-dev
·
for clarification, i'm not looking for a/b testing tools. rather, i'm looking for resources on how I can build these types of a/b tools myself. i have a high level overview of the data flow and architecture that i know my product should have. i need to hash each user_id to ensure each user consistently gets the same version, and also need to ensure that each version is split evenly among the entire userbase. then i should store this user_id-version mapping for quick lookup later. but i'm stumped when it comes to this:
- how do i know which hashing algorithm to use to ensure that each version is split evenly among the userbase? - is it better to build and run this hashing and allocation in-house or are there external services that help with this?
yen223 ·8 days ago
- The most useful resource we've found was from Spotify, of all places: https://engineering.atspotify.com/category/data-science/
- For hashing, md5 hash on (user-id + a/b-test-id) is sufficient. In practice we had no issues with split bias. You should not get too clever with hashing. You'll want to stick to something reliable and widely supported to make any post-experiment analysis easier. You definitely want to log the user-id to version mapping somewhere.
- As for in-house vs external, I would probably go in-house, though that depends on the system you're A/B testing. In practice the amount of work needed to integrate a third-party tool was roughly the same as building the platform, but building the platform meant we could test more bespoke features.
makingstuffs ·9 days ago
If it’s a tool you ever want to get into production you need to focus on which metrics you are collecting, how you’re collecting them, how you’re defining a session, how are you defining a user, how are you complying with data privacy laws, what tools and data platforms are you integrating with, what safeguards are you implementing to ensure that you’re not leaking sensitive information, how are you handling code delivery, how are you accounting for ad blockers… to name a few.
I’d personally look for some open source tools and assess what features they provide/how they provide them to get a good idea of what is required. Better yet, I’d try and get some trials on the current market leaders such as AB Tasty, Optimizely, Web Trends Optimize, Monetate and VWO.
You’ll notice that they all offer more than just A/B testing…
Show replies
codingdave ·9 days ago
But the work comes when analyzing the data later, so that is where you need to put in more thought. What are you measuring? How are you logging the events that will track that measurement? Those are both harder and more important questions. Based on what you are measuring, you might want only specific subsets of users to test things - why assign 50% of an entire userbase to a feature being tested that only 10% of the users touch? You need to assign a portion of that 10%, not a global 50%.
Basically, you need to start with the end data you intend to act on and work backwards, not start with a hashing algorithm that might not even help.
Show replies
AlchemistCamp ·9 days ago
Show replies
phyrex ·9 days ago
That's basically all you need for this. The hash algorithm doesn't matter as long as it's fast and returns a number.
For your example it's "hash(user_id) % 2" because a 50/50 split has two buckets.
Show replies