News for hackers. Stuff that matters.
Re:100 more of those BITFIELDs by derefr on Friday February 26, 2016 @02:26PM (#11183461)
With the #N syntax, Redis is getting dangerously close to having C-struct-typed fields (i.e. product types with a known bit-level encoding.)

I could imagine a STRUCT LOAD command (like SCRIPT LOAD) to push a struct definition to the server; and then STRUCT GETFIELD and STRUCT SETFIELD commands to manipulate the structs. The GETFIELD and SETFIELD commands would just boil down to the same BITFIELD command after dereferencing. (Heck, Redis could probably cheaply tag the fields with their struct definition, and then you could GET/SETBYFIELD would be polymorphic.)

...thinking about this more, this is possible now, isn't it? The struct-definition loader command, and the dereferencing commands, can all just be Lua functions passed to EVAL. The struct definitions could be loaded into a sorted set; the dereferencing commands would read from that set and then translate your query into a call to this BITFIELD command to do the work.

Anyone care to implement this silliness? :)

Re:100 more of those BITFIELDs by itamarhaber on Friday February 26, 2016 @02:55PM (#11183724)
Not that silly and perfectly doable with the current support for Lua scripts...
Re:100 more of those BITFIELDs by a1k0n on Friday February 26, 2016 @01:56PM (#11183224)
Check out Concise [0] or Roaring Bitmaps [1] before rolling your own compressed bitset scheme. Wish I had known about these sooner.

(I realize now this is offtopic, as this is bit fields not bit sets; nevermind, but still it's interesting!)

[0] http://arxiv.org/pdf/1004.0403.pdf

[1] http://roaringbitmap.org/

Re:100 more of those BITFIELDs by g4nt1 on Friday February 26, 2016 @02:16PM (#11183369)
+1 for Roaring Bitmaps. We are starting to use it in multiple products.
Re:100 more of those BITFIELDs by connor4312 on Friday February 26, 2016 @11:14AM (#11182138)
I'm quite excited for bitfields in Redis. We are going to need to implement some bloom filter-based data structures for one of our products in the near future. Previously the best way to do this was to either keep a hash map or a binary k/v, both of which have substantial drawbacks (such as the need for locking). These bitfields, however, are a perfectly-timed ideal solution to the problem.
Re:100 more of those BITFIELDs by dvirsky on Friday February 26, 2016 @01:50PM (#11183187)
You can implement bloom filters in redis >2.6 with SETBIT And GETBIT. But I'm guessing this is not your case exactly? I'm curious what exactly you're trying to model.
Re:100 more of those BITFIELDs by connor4312 on Friday February 26, 2016 @02:01PM (#11183263)
We want to count the number of users, including guests, who are currently viewing a resource. To count guests, we plan to count unique IPs, so each IP should be counted at most once even if it connects to several different servers as a result of load balancing.

The approach we're moving towards is an augmented bloom filter where each "bit" is actually an unsigned integer counting the number of items hashed to the position, allowing us to add and remove IPs to the filter as users come and go (just rehashing and "subtracting" the IP when it leaves). This lets us ensure that IPs are not counted more than once. SETBIT and GETBIT would work with a normal filter, but they can't increment multiple bits, so we'd end up needing to lock.

Re:100 more of those BITFIELDs by infogulch on Friday February 26, 2016 @03:00PM (#11183770)
Ah yes, a counting filter. [0] How do you plan on tackling rare overflow?

Have you looked at Cuckoo Filter [1]? Same abilities as a counting bloom filter, but has some advantages like reducing false positives while using even less memory and requiring less processing.

[0]: https://en.wikipedia.org/wiki/Bloom_filter#Extensions_and_ap... [1]: https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf

Re:100 more of those BITFIELDs by antirez on Friday February 26, 2016 @02:54PM (#11183712)
I wonder if you could use Redis hyperloglogs for this... You count IP ins in an HLL, COUNT IP out, and subtract.
Re:100 more of those BITFIELDs by dvirsky on Friday February 26, 2016 @04:39PM (#11184473)
sounds to me like it's more suitable for HLL than for blooms actually.
Re:100 more of those BITFIELDs by andrewfromx on Friday February 26, 2016 @11:13AM (#11182122)
i love using 64 bit unsigned ints in mysql for 64 boolean columns. I do:

alter table foo add column bar bigint(20) unsigned default 18446744073709551615

and that gets you 64 111111111111111111111111111111 etc.

Re:100 more of those BITFIELDs by Zikes on Friday February 26, 2016 @11:19AM (#11182170)
That sounds pretty painful to query, though. If the first bit position represented if the user had confirmed their email address, wouldn't a SELECT for confirmed users look something like SELECT * FROM users WHERE (mydata & b'0000000000000000000000000000000000000000000000000000000000000001')?
Re:100 more of those BITFIELDs by 33degrees on Friday February 26, 2016 @12:10PM (#11182509)
No, you can just compare against an integer: SELECT * FROM users WHERE mysql & 1
Re:100 more of those BITFIELDs by astrange on Saturday February 27, 2016 @12:16AM (#11186176)
This sounds like it'd be a full table scan to me.
Re:100 more of those BITFIELDs by andrewfromx on Friday February 26, 2016 @11:21AM (#11182193)
in rails I add:

  [:foo_boolean1, :foo_boolean2].each_with_index do |flavor, i|
    power = 1 
    i.times do
      power *= 2
    end 
    define_method "has_#{flavor}" do
      (foo_options & power) > 0 
    end 
    define_method "has_#{flavor}=" do |value|
      value = value == '1' if value.class == String
      current_value = self.send("has_#{flavor}")
      if (current_value && !value) || (!current_value && value)
        self.foo_options = foo_options ^ power 
      end 
    end 
  end 
which lets me access each col like normal. blah.has_foo_boolean1 or blah.has_foo_boolean1=true etc.
Re:100 more of those BITFIELDs by 33degrees on Friday February 26, 2016 @12:02PM (#11182459)
You might want to take a look at https://github.com/pboling/flag_shih_tzu, which does all of that and a lot more, like adding database scopes.
Re:100 more of those BITFIELDs by andrewfromx on Friday February 26, 2016 @12:18PM (#11182578)
reminds me of this http://jeremykun.com/2013/02/08/why-there-is-no-hitchhikers-... article about doing Math cuz u love it.
Re:100 more of those BITFIELDs by andrewfromx on Friday February 26, 2016 @12:17PM (#11182573)
aw, but it was so much fun writing that logic myself and really learning how bit ops work in ruby.
Re:100 more of those BITFIELDs by andrewfromx on Friday February 26, 2016 @11:14AM (#11182133)
so much better than individual boolean columns cluttering up your user table.
Re:100 more of those BITFIELDs by kbenson on Friday February 26, 2016 @12:32PM (#11182673)
Poe's law in action.

I can't tell if you are serious, but if you are, there are generally database constructs to help with this (if the database doesn't just outright support condensing bit fields or a specific type for sets) in a way that doesn't make it impossible to read easily from the database. You're basically making your data unusable without the separate code you've written to parse that field, which can be really annoying in many, many instances (such as debugging!).

Edit: It's worth noting that one way to use a complex structure in the database without making it too dependent on external code would be to create some stored procedures for setting, updating and viewing the structure. In this case, that would be some functions that took the field value, and extracted the individual bits into a result, or allowed updating a specific flag. That would keep how the data is represented within the purview of the database, but you're stuck with a bunch of SQL functions that approximate what your code is doing in the application, which is a pain. Thankfully, a lot of database systems acknowledged this a long time ago and provided constructs to make it easier.

Re:100 more of those BITFIELDs by Dylan16807 on Friday February 26, 2016 @01:45PM (#11183152)
53 bits.. they're returning a bitfield as a double float at some layer?

Or am I misreading the implication and they can do signed 64 bits but not unsigned?

Re:100 more of those BITFIELDs by bluedino on Friday February 26, 2016 @12:25PM (#11182614)
Would like to see speed vs storage benchmarks. Bit twiddling must be a bit slow, no?
Re:100 more of those BITFIELDs by dxhdr on Friday February 26, 2016 @01:43PM (#11183141)
The only slow operations these days are L3 cache misses and transcendental function calls, around 100 cycles each. Optimize for that, not a 1 CPU clock cycle "bit twiddle."
Re:100 more of those BITFIELDs by JoachimSchipper on Sunday February 28, 2016 @10:18AM (#11191075)
Computers are generally very good at bit-twiddling - why do you expect this to be slow?