News for hackers. Stuff that matters.
Re:Mimic – abusing Unicode to create tragedy by mattlondon on Friday October 23, 2015 @07:24AM (#10437901)
I had something similar happen in the wild to me.

I work for a "major search engine" that does a lot of advertising & marketing stuff. To get the most out of it, we need customers to implement some javascript on their ecommerce sites.

As is often the case, javascript code that needs to get implemented on an ecommerce site often gets copy-pasted or emailed around a lot internally within a customer before it reaches the right person who can add it to the site's pages.

In this example somewhere along the way, a normal javascript snippet got all of the semi-colons changed from ; to ;.

In case you've not already spotted it, ; is not a ; but is actually "Greek Question Mark" (http://www.fileformat.info/info/unicode/char/037e/index.htm).

It was very confusing why Chrome was moaning about a semi-colon an illegal token. I had a genuine "Am I going mad? Seriously?" moment before I realised what was happening.

Re:Mimic – abusing Unicode to create tragedy by podgib on Friday October 23, 2015 @08:19AM (#10438101)
I've been bitten by things like this so many times, that my first reaction on seeing an inexplicable syntax error is to delete and manually re-type the line.
Re:Mimic – abusing Unicode to create tragedy by TazeTSchnitzel on Friday October 23, 2015 @12:18PM (#10439541)
I was recently bitten by OS X's non-breaking space shortcut. I'd accidentally typed alt+space instead of space due to typing a following # (alt+3), and that made weechat, an app unaware of the NBSP, see the command '/join #channel' instead of '/join'.
Re:Mimic – abusing Unicode to create tragedy by slowmotiony on Friday October 23, 2015 @07:38AM (#10437942)
I would probably have to quit my job before I could figure out that problem. May I ask how did you spot it?
Re:Mimic – abusing Unicode to create tragedy by mattlondon on Friday October 23, 2015 @08:59AM (#10438266)
It was a while ago now, but I remember copying the code out to a controlled isolated environment on my local machine, picking the first line and making sure it reproduced.

After that it was a matter of just going through the usual steps of working out why something isn't working. I think in this instance I actually ended up manually retyping the code, and when getting "identical" code that worked that was when the penny dropped and I realised something was not what it seemed. If you paste suspect characters into something like http://unicode-table.com/en/ it will tell you right away what it really is.

Re:Mimic – abusing Unicode to create tragedy by austinjp on Friday October 23, 2015 @08:48AM (#10438223)
Weird bugs can respond well to brutally dumb debugging. Open file and search for ; perhaps? Or grep etc. When you can see semi-colons but your find tool can't, that's a worry.

But yeah getting to that point relies on pure inspiration, in my case.

Re:Mimic – abusing Unicode to create tragedy by err4nt on Friday October 23, 2015 @09:15AM (#10438343)
I find searching, and search and replace the most useful IDE feature, you can reformat entire documents with a series of well thought out search and replaces:)
Re:Mimic – abusing Unicode to create tragedy by falcolas on Friday October 23, 2015 @08:50AM (#10438231)
Personally, my vim status line has an indicator which shows the hex code for the rune currently under the cursor.

After fighting against word processor quotes, it's become second nature to double check it periodically.

Re:Mimic – abusing Unicode to create tragedy by brunosutic on Friday October 23, 2015 @12:10PM (#10439496)
I just created a vim plugin "vim-troll-stopper" https://github.com/vim-utils/vim-troll-stopper

It protects you from these tricks by highlighting "troll" unicode characters in red.

Re:Mimic – abusing Unicode to create tragedy by kps on Friday October 23, 2015 @10:41AM (#10438840)
How about,

  :syn match Error "[^ -~]"
Re:Mimic – abusing Unicode to create tragedy by iamcurious on Friday October 23, 2015 @09:30AM (#10438388)
That seems useful. Can you share it?
Re:Mimic – abusing Unicode to create tragedy by falcolas on Friday October 23, 2015 @09:45AM (#10438467)
Here you go:

    :set statusline=%F%m%r%h%w\ [TYPE=%Y]\ [ASCII=\%03.3b]\ [HEX=\%02.2B]\ [POS=%04l,%04v][%p%%]\ [LEN=%L]
    :set laststatus=2
Produces a status line when inserting and recording a macro like (the character under the cursor is 'm':

    ~/.vimrc [TYPE=VIM] [ASCII=109] [HEX=6D] [POS=0123,0020][67%] [LEN=182]
    -- INSERT --recording
And, of course... `:help statusline`
Re:Mimic – abusing Unicode to create tragedy by rspeer on Friday October 23, 2015 @11:29AM (#10439202)
This doesn't seem Unicode-aware at all. I put my cursor on the character 每 and it says:

  [ASCII=2>4] [HEX=0>4]
It's not ASCII, and its hex code is 6BCF.

It also says "ASCII=252" when I put the cursor over "ü". Claiming that values over 127 are ASCII is just a malapropism.

Re:Mimic – abusing Unicode to create tragedy by Procrastes on Friday October 23, 2015 @11:32AM (#10439226)
vi can be so elegant.

For us emacsen you can do a (what-cursor-position &optional DETAIL) which is usually bound to Ctl-x =

I don't think it will be nearly this clean to add it to the modeline, but I'll take a look.

Re:Mimic – abusing Unicode to create tragedy by iamcurious on Friday October 23, 2015 @03:05PM (#10440641)
Very nice! Thanks.
Re:Mimic – abusing Unicode to create tragedy by coldpie on Friday October 23, 2015 @09:29AM (#10438387)
Cool! How do you do that?
Re:Mimic – abusing Unicode to create tragedy by leephillips on Friday October 23, 2015 @09:44AM (#10438462)
I'd like to know, too. Until then, you can type "ga" in normal mode to get a display of the decimal, hex, and octal value of the character under the cursor.
Re:Mimic – abusing Unicode to create tragedy by CUViper on Friday October 23, 2015 @04:52PM (#10441230)
vim-characterize has a nice enhancement of "ga", adding Unicode info, digraphs, emojis, and HTML entities. https://github.com/tpope/vim-characterize
Re:Mimic – abusing Unicode to create tragedy by falcolas on Friday October 23, 2015 @09:46AM (#10438473)
Replied with it in a sibling comment.
Re:Mimic – abusing Unicode to create tragedy by slig on Friday October 23, 2015 @07:42AM (#10437953)
Something similar happen to me, but with that fancy quotes. I spotted by doing a "binary-search weird bug hunt". Cut half of the code off and see it it's still complaining, if it's, cut the other half, and so on.
Re:Mimic – abusing Unicode to create tragedy by jprince on Friday October 23, 2015 @08:49AM (#10438228)
Ah, that old stand-by. Always warm and ready for the odd unicode hell-bug.
Re:Mimic – abusing Unicode to create tragedy by sethammons on Friday October 23, 2015 @07:58AM (#10438014)
I had a similar problem. Copied a code snippet. Ruby started complaining about an undefined function. After nearly going mad, and then looking at the source through a hex editor, you could see Unicode whitespace. I have yet to forgive ruby or Unicode whitespace. Or the chat utility from whence I copied.