orcadk
SQL Server Corruption Recovery - When All Else Fails
The message is: Nobody is allowed to report misconduct in this way.
Furthermore, the web has been flooded with the exact same animation since the midst of 2004, all implemented in Flash. In the later years, these have also been implemented in pure HTML: https://github.com/blasten/turn.js
Given that Apples patent wasn't filed for until December 2011, I can't see why it wouldn't be dismissed on the grounds of prior art being publicly available.
Apple wins design patents for page turning animation and iPad Smart Case
Blizzard has already executed this impressively for several of their titles. Starcraft II is an 8GB download, but after about 400MB you can actually start the game. It'll then continue downloading non-invasively in the background while loading needed data on demand. Same goes for World of Warcraft, and probably other titles as well.
First up, in his example we're talking about 10k rows with a size of 35 bytes. At 8050 bytes of available space per page, that gives a fan-out of 8050/35 = 230. 10k/230*8kb = 350kb. At 350kb of inserts (hobt data, he doesn't go into secondary indexes), memory is completely irrelevant - the only force going on here is latency on writing the log to disk.
If we had a huge data set (as in, did not fit in memory, at all) with high cardinality - sure, we'd have a lot of cold leaf level pages. With no further info on his case, I can only assume most of the hobt and secondary indexes will fit in memory. At worst we'll have to read a cold leaf level page into memory to perform the addition in-memory.
As is there's no mention of even a clustered index, causing all of this to be heap inserts which is arguably one of the fastest insert methods there are (barring certain very special cases).
First up, the non-SQL-injection code doesn't specify the parameter types. As such, each length of monster_class will result in a different execution plan being compiled and cached by SQL Server. Granted, there's only 50 different kinds - but just specify the parameter types to avoid it.
Sure, you could setup a VLAN, but we're lightyears from having that affect our performance at this point.
Parser overhead is not an issue as long as parameter types are specified. The plan will be compiled and cached. The only advantage to stored procedures at this point is reduced network data as well as management.
Page faults - Sure, but for inserts the data will most likely be in memory already as we're writing new data. And if there's enough data, the relevant b-tree pages will most certainly be hot.
Varchars - Granted, avoid them unless you're storing string data. I have no idea where the "hundreds of bytes of of memory" comes from. SQL Server caches all data as pages, exact copies of what's stored on disk. Variable-length varchars have an overhead of exactly two bytes + 2/#varlengthCOlumns in total.
Disk IO - The transaction log is never merged with the main data file. SQL Server uses write ahead logging, writing to the log first while modifying the data pages in memory. Now and then a checkpoint is run, writing all the dirty pages to the main data file. A log is only "merged" with the data file if the server has crashed and recovery is being run automatically during startup.
"This increases the number of rows you can insert before SQL Server succumbs to memory pressure" - Since the article only deals with inserts, why are we talking about memory pressure? You'll get disk contention for sure, but memory won't realistically be an issue unless you're reading as well.
There's an alternative to keeping the potentially long running transaction open - avoiding both SqlBulkCopy and transferring datatables. The BCL has a CommandSet notion that utilitizes a TDS feature that allows batches to be streamed to SQL Server. Unfortunately you'll have to use reflection to get at it (https://github.com/ayende/rhino-etl/blob/master/Rhino.Etl.Co...), but the code has been stable for several releases and is very simple to interface. It basically allows you to create normal SqlCommands and just send a large batch of them at once, all being executed atomically with minimal locking.
Finally, with the micro optimizations being mentioned, I'm surprised he doesn't recommend to "SET NOCOUNT OFF".
Overall, some good points with some inaccuracies. I have a hard time with the closing statement "As you can see, you don’t need to abandon SQL Server to get massive performance improvements.". These are decent improvements, but this is such a simple scenario that I find it pretty optimistic to let this define whether to abandon SQL Server or not. The RDBMS problem is rarely with inserting enough data, but with reading data at the same time while avoiding locking.
His tag line is "Denmarks big show-off" and he basically markets himself as a provocative bragging show-off, always with the rolexes, sports cars, slick hair and better-than-thy appearance.
He's not perceived as a comedian but rather as an entertainer hired for various corporate events - some obviously less successful than others.
Unless this was booked by Dell US without doing any kind of checkup, they would've been aware what would be coming. He's knowing for his provocative presentations and I have a hard time seeing exactly how a Danish Dell branch would find it appropriate or interesting to book Mads Christensen. If they wanted stand-up comedy we've got lots of skilled comedians. Mads Christensen is not in that line of work.
To be fair I don't think Mads Christensen personally believes what he presents on stage. It's the persona he's built up. Not that that makes it better in this scenario, just so flak is directed where it's most appropriate - Dell, and especially whoever made the decision to book him.
"For your security we do NOT store your IP address/result combination"
So the IP/result combination is not stored, but technically the IP address and result could be stored without there being a relation between them?
However, you can create columnstore indexes on specific partitions, and thus, by partitioning your source data, you can continue to add new data in new partitions. As most DW/"big data" solutions use partitioning anyways, this is a usable solution, giving you the benefits of the readonly performance as well as the maintainability aspects, while still allowing you to feed in new data.
Columnstore indexes are not meant for OLTP type solutions, so there's no need for realtime regular data deletion; it simply doesn't happen on this kind of historical data. You might switch out a partition of your data, or you might switch in a partition of new data, but you won't change the actual data itself.
Comparing SQL Servers columnstore index implementation with other columnstore based databases doesn't make sense. I won't laud columnstore indexes as a revolutionary new feature, but the way it's implemented is new, AFAIK. Comparing it to "normal" columnstore based databases will make SQL Server look bad - but the thing is, it's not meant to compete with those. Use it where it makes sense.
Creating Multiplexers Using Logic Gates
RAID5 is notoriously bad for write performance, just as RAID6 is. This is a classic writeup on why you should avoid RAID5 at all costs for a scenario like this: http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt
Adding 6+7 At The Logic Gate Level
With such a common use of them, we can start looking at how to optimize the catalog format itself, rather than stick with the scrolling vs. catalog format discussion.
To start out - I hate the pageflip effect. In the beginning, clients would drop the jaw to the floor when they saw the effect. They'd go into a coma when they saw you could drag the corners. We didn't have to do any sales beyond that - the effect alone sold it and the catalogs spread like wildfire. Sorry.
As time progressed, the pageflip effect alone didn't really cut it, especially as competition appeared. Since then we, as others, have gone in different directions, usually in the form of offering different addon modules, integration, etc. As time went on, the effect got diluted as well - we no longer offer the user the ability to actually drag the corners - usability testing proved that to be bad and cumbersome to use, instead you only get the option to flip the page by click big "Next/Previous" bars next to the catalogs. The effect is only seen while the pages flip automatically after an arrow click.
So why use the format at all? As many others say, normal scrolling is much more web friendly. However, there are billions of catalogs out there today, and millions more produced each and every day. Companies have hundreds of thousands of product manuals lying around, and they'd like to show them in a lightweight format that doesn't require you to download the 200mb PDF.
There are many retail chains that still send physical catalogs to your physical mail boxes each and every day. These catalogs are usually produced by print agencies on a weekly basis, and to also produce the same content in a webfriendly manner, that'd require tonnes of extra work. Instead, they can utilize what they're already producing and just put it online in an interesting format to read, for the common user.
To sum up - the flip effect was overused, but has now found a decent niche where it can be used. Lots of material warrant the format, even though it's not the optimal one.
I think turn.js looks very interesting. I have a boatload of catalogs running on Flash today, and I'd love to change that.