Hi yeldarb! This is a really neat app. I don't want to diminish your creation, so please don't think I'm trying to put you down with this question:

How much of this was just glue-code to integrate a few modules, vs. how much did you have to do from scratch?

I'm asking to try to get an idea to see if ARkit is within the reach of an average joe- sudoku solver code could be a copy/paste, digit recognition could be a copy/paste, and ARKit should provide you with some environment/location info.

It's mostly glue (but some of it is special glue).

Oddly enough, digit recognition was one of the harder parts. iOS11's Vision API has character recognition but it only gives you a bounding box, it doesn't do the OCR!

So for that I ended up ripping up a whole bunch of Sudoku puzzle books from the second hand book store and scanning them in to create a dataset. Then I trained my own nerual network in Keras and converted it to CoreML.

The actual ARKit part was really easy, even for me who has never done iOS programming before.

Have you tried to render individual digits in the existing empty squares instead of rendering a whole new grid? Coupled with a font detection algorithm this could end up looking more magical.

Originally that was the goal. Unfortunately it wouldn't align exactly perfect (either the size was slightly off due to whether the rectangle detection snagged the inside or outside of the borders or the depth was slightly off so there was slippage when moving around).

Didn't quite work right.

I do have some things on my to-do list to experiment with though that might help the magic

Hey, congrats on shipping! Unfortunately your app doesn't work for me :(. I've tried it on few sudoku designs on my laptop screen (don't have anything on hand right now) and there were two problems - the camera would not focus on the screen which resulted in blurry image, and second - it didn't recognize the puzzle correctly and returned incorrect results. I did put my screen flat on the table to simulate a paper laying on it, because I noticed the app doesn't work when the screen was upright - it rendered the results horizontally anyway.

Thanks for giving it a shot! Sorry it's not working for you.

I'm working on a hack to get non-horizontal puzzles working right now.

Unfortunately ARKit can only detect horizontal planes at this point so I'm trying to find a workaround since I now realize most people won't have a paper sudoku laying around to try it out and will logically go to the web to find one.

I suspect what's happening with the bad reads is that my neural network for digit classification is trained only on paper puzzles (I chopped up a bunch of books and scanned them to make my dataset). I'll have to get some scans of computer screen puzzles to get better accuracy.

Can you let me know what site you tried to use so I can make sure to get that font included in the dataset as well?

Hoping to get v1.1 submitted tonight yet that will fix the horizontal plane deficiency. Collecting, labeling, and retraining the network may take a bit longer than that.

Your app is awesome! What type of neural network did you use? Please write a post, I’m teaching myself machine learning and iOS development, and would love to learn from a successful developer!

Thanks! It's using a convolutional neural network which I minimally adapted from an MNIST tutorial (one of the reasons I picked this project to start with is that there are so many MNIST tutorials for how to do digit recognition out there and it seemed like a good way to dip my toe into ML).

Converting it to CoreML after it was trained was really easy.

The hardest part was actually creating the dataset and wrestling with pixel buffers to get the image data into a format CoreML liked.

Just tried it out, and it looks very good. It was a perfect way to demonstrate ARKit to colleagues. But for some reason it flips the board upside down.

Sorry about that!

That was happening when the phone traveled to the other side of the scene's origin from where it started. The angle calculation got confused.

I just got that fixed and submitted to Apple in v1.1 this morning. Should be inbound today or tomorrow as soon as they approve it.

Were there any particular advantages in using a neural network to solve the sudoku as opposed to an existing algorithm?

Actually the solving of the puzzle itself is just done with a simple recursive algorithm.

I trained a neural network to do the computer vision part to interpret the puzzles.

And there are a bunch of pre-trained networks built into iOS11's Vision API and ARKit that the app just uses black box.

ahh, sorry for the misunderstanding.

Does the network do well with handwriting, e.g. puzzles that have been half-solved?

Well I wasn't going to spill the beans just yet but if you promise to keep a secret.. working on that for v2 :)

Edit: Although so far I've learned that some people have VERY bad handwriting. I'm having a hard time sanitizing a dataset because I can't even tell what digit they were trying to write

Do the standard datasets not work too well (e.g. MNIST or NIST)?

http://yann.lecun.com/exdb/mnist/

http://www.cs.toronto.edu/~tijmen/affNIST/

https://www.nist.gov/srd/nist-special-database-19

Tried those first but no, they didn't work great with the data I was able to extract; those datasets are only handwritten digits which apparently weren't close enough to the computer printed ones.

I think the artifacts from the square borders (eg if it's slightly mis-cropped or misaligned) were also tripping up the model before I had my custom dataset loaded in there.

Edit: I should say this was my first stab at doing machine learning so I may be missing something; definitely open to ideas or suggestions!

Shouldn't you be able to generate a lot of synthetic "printed-looking" digits very easily? Just get say 20 common fonts and render each digit with random rotations and scaling etc.

That's definitely on my list of things to try! Thanks for the push in that direction

I'm that guy. Sorry :(

I have the worst handwriting.

People are like, "Dude, wtf?" when I try to whiteboard stuff.

All good; validates that I'm at least on the right track! Had to pare back my ambitions quite a bit to get _something_ finished before the iOS11 launch.

I didn't expect the reception to be so enthusiastic for v1 tbh; but I guess I chose a good place to call it an "MVP"

I'm really interested in the growth of ML on mobile apps. Did you test the ability to call existing ML APIs in the cloud and found they weren't responsive enough, or did it just feel more natural to run the net locally?

I've played with AWS Rekognition a little bit. It's definitely WAY faster to do it on-device. Plus it works even in Airplane mode.

I'm running the iOS11 Vision stuff, my own text recognition CoreML neural network (81 times per loop), plus all the ARKit stuff and I can still get 30-60 fps on my iPhone 7+

There shouldn’t be a need for that. All the info information you need to solve it all come with the puzzle.

They want to recognize the handwriting to give the player feedback on errors made.

Are you the same yeldarb as the one from kirupaForum?

I am! Good to see you :)

I miss that place; I feel like we grew up there.

Awesome! It's surreal to see a familiar name making such a neat app. I saw this earlier today and had a small moment of triumph since I'm terrible at solving sudokus myself. A few people visit kF every once and a while; feel free to stop by!