The need for efficient and concise URL management has become important in today’s digital world. URL shortening services help convert long and complex links into short, easy-to-share URLs. This improves usability, especially on social media and messaging platforms.
Short URLs are simple to share and remember.
Helps track clicks and manage links effectively.
Example: A long e-commerce product link can be shortened into something like bit.ly/xyz123, making it easier to share on platforms like WhatsApp or Twitter.
1. System Requirements
These requirements define what the system should do and how well it should perform.
1. Functional requirements
These describe the core features that the system must provide to users.
Given a long URL, the service should generate a shorter and unique alias for it.
When the user hits a short link, the service should redirect to the original link.
Links will expire after a standard default time span.
2. Non-Functional requirements
These define the performance, reliability, and security aspects of the system.
The system should be highly available. This is really important to consider because if the service goes down, all the URL redirection will start failing.
URL redirection should happen in real-time with minimal latency.
Shortened links should not be predictable.
2. Capacity Estimation
Let's assume our service has 30M new URL shortenings per month. Let’s assume we store every URL shortening request (and associated shortened link) for 5 years. For this period the service will generate about 1.8 B records.
30 million * 5 years * 12 months = 1.8B
Note: Let's consider we are using 7 characters to generate a short URL. These characters are a combination of 62 characters [A-Z, a-z, 0-9] something like https://zpzy.in/redirecting?code=abXdef2.
Data Capacity Modeling
Discuss the data capacity model to estimate the storage of the system. We need to understand how much data we might have to insert into our system. Think about the different columns or attributes that will be stored in our database and calculate the storage of data for five years. Let's make the assumption given below for different attributes.
Consider the average long URL size of 2KB ie for 2048 characters.
Short URL size: 17 Bytes for 17 characters
created_at- 7 bytes
expiration_length_in_minutes -7 bytes
The above calculation will give a total of 2.031KB per shortened URL entry in the database. If we calculate the total storage then for 30 M active users total size = 30000000 * 2.031 = 60780000 KB = 60.78 GB per month. In a Year of 0.7284 TB and in 5 years 3.642 TB of data.
Note: We need to think about the reads and writes that will happen on our system for this amount of data. This will decide what kind of database (RDBMS or NoSQL) we need to use.
3. Low-Level Design
In the context of a TinyURL system, Low-Level Design focuses on how individual components like URL generation, storage, and redirection are implemented. It defines classes, methods, and database structure to efficiently create, store, and retrieve shortened URLs.
LLD
URL Encoding Techniques to create Shortened URL
To convert a long URL into a unique short URL we can use some hashing techniques like Base62 or MD5. We will discuss both approaches.
1. Base62 Encoding
Base62 encoder allows us to use the combination of characters and numbers which contains A-Z, a-z, 0–9 total( 26 + 26 + 10 = 62).
So for 7 characters short URL, we can serve 62^7 ~= 3500 billion URLs which is quite enough in comparison to base10 (base10 only contains numbers 0-9 so you will get only 10M combinations).
We can generate a random number for the given long URL and convert it to base62 and use the hash as a short URL id.
If we use base62 making the assumption that the service is generating 1000 tiny URLs/sec then it will take 110 years to exhaust this 3500 billion combination.
MD5 also gives base62 output but the MD5 hash gives a lengthy output which is more than 7 characters.
MD5 hash generates 128-bit long output so out of 128 bits we will take 43 bits to generate a tiny URL of 7 characters.
MD5 can create a lot of collisions. For two or many different long URL inputs we may get the same unique id for a short URL and that could cause data corruption.
So we need to perform some checks to ensure that this unique id doesn't exist in the database already.
Efficient Database Storage & Retrieval of TinyURL
Let's discuss the mapping of a long URL into a short URL in our database:
1. Using Base62 Encoding
Assume we generate the Tiny URL using base62 encoding then we need to perform the steps given below:
The tiny URL should be unique so firstly check the existence of this tiny URL in the database (doing get(tiny) on DB). If it's already present there for some other long URL then generate a new short URL.
If the short URL isn’t present in DB then put the long URL and TinyURL in DB (put(TinyURL, long URL)).
This technique works with one server very well but if there will be multiple servers then this technique will create a race condition.
When multiple servers will work together, there will be a possibility that they all can generate the same unique id or the same tiny URL for different long URLs.
Even after checking the database, they will be allowed to insert the same tiny URLs simultaneously in the database and this may end up corrupting the data.
2.Using MD5 Approach
Encode the long URL using the MD5 approach and take only the first 7 chars to generate TinyURL.
The first 7 characters could be the same for different long URLs so check the DB (as we have discussed in Technique 1) to verify that TinyURL is not used already.
This approach saves some space in the database
If two users want to generate a tiny URL for the same long URL then the first technique will generate two random numbers and it requires two rows in the database.
In the second technique, both the longer URL will have the same MD5 so it will have the same first 43 bits.
This means we will get some deduping and we will end up with saving some space since we only need to store one row instead of two rows in the database.
3. Using Counter Approach
Using a counter is a good decision for a scalable solution because counters always get incremented so we can get a new value for every new request.
Single server approach:
A single host or server (say database) will be responsible for maintaining the counter.
When the worker host receives a request it talks to the counter host, which returns a unique number and increments the counter. When the next request comes the counter host again returns the unique number and this goes on.
Every worker host gets a unique number which is used to generate TinyURL.
4. High-level Design
High-Level Design defines the overall architecture of a URL shortening system, including components like API servers, databases, and caching layers. It shows how these components interact to generate short URLs and handle redirection efficiently.
HLD
1. User Interface / Clients
Users enter a long URL via web form or API and receive a shortened link.
Users input long URLs
System returns short URLs
2. Application Server
Handles core logic like key generation, storage, and redirection.
Generates short keys and stores mappings
Manages redirects and tracking
3. Load Balancer
Distributes incoming traffic across multiple system components.
Balances load across application servers
Distributes traffic to database and cache
4. Database
Stores persistent URL mappings and ensures scalability.
Stores URL–key mappings
Uses NoSQL systems like MongoDB or Cassandra
5. Caching
Improves performance by storing frequently accessed data.
Stores popular URLs in memory
Uses Redis or Memcached
6. Cleanup Service
Maintains system efficiency by removing unused data.
Deletes expired URLs
Frees up storage
7. Redirection
Handles user requests by mapping short URLs to original URLs.
Retrieves original URL
Redirects using HTTP 301
8. Analytics
Tracks and analyzes user interactions.
Tracks clicks and user data
Provides usage insights
9. Security
Protects the system from malicious activities.
Prevents phishing and DDoS attacks
Uses authentication and rate limiting
5. Database Design
Explore some of the choices for System Design of Databases of URL Shortner:
We can use RDBMS which uses ACID properties but you will be facing the scalability issue with relational databases.
Now if you think you can use sharding and resolve the scalability issue in RDBMS then that will increase the complexity of the system.
There are 30M active users so there will be conversions and a lot of Short URL resolution and redirections.
Read and write will be heavy for these 30M users so scaling the RDBMS using shard will increase the complexity of the design.
You may have to use consistent hashing to balance the traffic and DB queries in the case of RDBMS and which is a complicated process. So to handle this amount of huge traffic on our system relational databases are not fit and also it won't be a good decision to scale the RDBMS.
We write something and it takes some time to replicate to a different node but our system needs high availability and NoSQL fits this requirement.
NoSQL can easily handle the 30M of active users and it is easy to scale. We just need to keep adding the nodes when we want to expand the storage.
6. Caching and Load Balancing in URL Shortening service
Caching and load balancing play a critical role in handling high traffic and ensuring fast response times in a URL shortening system. They help reduce database load and distribute user requests efficiently across servers.
1. Caching
Caching is used to store frequently accessed URL mappings so that redirection can happen quickly without hitting the database every time.
Read-Through Cache: Automatically fetches data from the database and stores it in cache when a cache miss occurs.
Write-Through Cache: Updates both the cache and database simultaneously whenever data is written.
Tools like Redis or Memcached are commonly used to improve performance.
2. Load Balancing
Load balancing distributes incoming traffic across multiple servers to ensure high availability and reliability.
Round Robin: Distributes requests evenly across all servers; simple and effective.
Least Connections: Sends requests to the server with the fewest active connections; better for uneven workloads.
