a phone number is not a 64 bit integer, like, just off the top of my head, a phone number can start with "0"
{
"message_id": "b72f9a6c-34d2-4ef9-89a5-623c1d7b890a",
"timestamp_sent": "2025-06-05T23:57:34Z",
"timestamp_received": "2025-06-05T23:57:35Z",
"sender": {
"phone_number": "+1-318-555-1234",
"carrier": "AT&T",
"device_id": "IMEI-354812345678901",
"location": {
"cell_tower_id": "LA5321",
"latitude": 32.5385,
"longitude": -92.9222
}
},
"receiver": {
"phone_number": "+1-225-555-5678",
"carrier": "Verizon",
"device_id": "IMEI-869712345678902",
"location": {
"cell_tower_id": "LA6723",
"latitude": 30.4515,
"longitude": -91.1871
}
},
"network": {
"protocol": "GSM",
"message_type": "SMS",
"message_size": "160 bytes",
"sms_center": "+1-800-555-9876",
"routing_path": [
"Cell Tower LA5321",
"Switching Center AT&T Baton Rouge",
"SMS Center",
"Switching Center Verizon New Orleans",
"Cell Tower LA6723"
]
},
"status": "Delivered"
}
get a celltower snooper on your phone and watch the data it shows - that's the metadata for your phone. SMS dragnet would need that for both phones, plus the message itself.
It's not an integer. But you can store it inside 64 bits. You can split it into country code and then number, or you can use 60 bits to store 18 digits and then use the top 4 bits to say how many leading 0s to keep/remove. Or other things. A 64 bit integer is enough bits to store variable length numbers up to 19 digits while remembering how many leading zeros they have.
If you want really simple and extremely fast to decode you can use BCD to store up to 16 digits and pad it with F nibbles.
> JSON
Most of this is unimportant. Routing path, really? And we don't need to store the location of a cell tower ten million times, we can have a central listing of cell towers.
I don't think we really need both phone number and IMEI but fine let's add it. Two IMEI means another 16 bytes. And two timestamps sure.
Phone number, IMEI, timestamp, cell tower ID, all times two. That's still well under 100 bytes if we put even the slightest effort into using a binary format.
> and again - if you use clever tricks to reduce this, you increase the overhead to actually use the data.
No no no. Most of the things I would do are faster than JSON.
How did you reach that number?
I figure the most important metadata is source and destination phone numbers and a timestamp, and I guess what cell tower each phone was on. A phone number needs 8 bytes, and timestamp and cell tower can be 4 bytes, so that's 28 bytes of important metadata.
> (quick math shows you went off something like ~32TB of sms data per 1e12 messages)
I was going for a full 140TB of data. 20-30TB hard drives are available.
I did consider metadata, but I figured you could probably put that in the savings from non-full-length messages.
> Where are you going to put the 1PB (for 1e12, but your own cite says it's 1e13, so 10PB) of metadata, minimum?
Well for just the US it would be closer to 1PB. But, uh, I'd store it in a single server rack? (ideally with backups somewhere) As of backblaze's last storage pod post, almost three years ago, it cost them $20k per petabyte. That's absolutely trivial on the scale of telecomms or governments or whatever.
> My argument is that: we need much more "metadata" than 1KB per meter to "know the road baseline, current conditions, hazards", much in the same way a text message is more than 140 bytes.
I mean, I agree with you about needing extra information.
But that's why the number I gave is 10000x larger than your CSV. My number is supposed to be big enough to include those things!
> note: the metadata for a meter of road could be:
I really appreciate the effort you put into this. I have two main things to say.
A) That's less than a kilobyte of information. Most of the bytes in the JSON are key names, and even without a schema for good compression, you can replace key names with 2-byte identifier numbers. And things like "critical" and "Active roadwork zone with lane closure" should also be 1-byte or 2-byte indexes into a table. And all the numbers in there could be stored as 4 byte values. Apply all that and it goes down below 300 bytes. If you had a special schema for this, it would be even lower by a significant amount.
B) Most of those values would not need to be repeated per meter. Add one byte to each hazard to say how long it lasts, 0-255 meters, instant 99% savings on storing hazard data.
> each "trick" you do to reduce the size of the metadata necessarily implies more CPU needed to parse and process it.
CPUs are measured in billions of cycles per second. They can handle some lookup tables and basic level compression easily. Hell, these keys are just going to feed into a lookup table anyway, using integers makes it faster. And not repeating unchanged sections makes it a lot faster.