Boogie Board Sync review

Simone Leave a comment

I was looking for a tool to replace all the paper clutter on my desk.

I’ve been looking high and low, and as of December 2018, to my knowledge these are the options if you want to hand-write your notes to an electronic device:

  • a iPad Pro or a Surface tablet, with an Apple Pen or the Surface Pen. They are Not Cheap (around ~1000€). I’ve only tried the Surface, and the input lag is noticeable.
  • a Samsung Android tablet with the S Pen, which, again, is Not Cheap (~700€).
  • another cheaper Android tablet with a regular capacitive pen, but the input lag is huge, and precision is poor.
  • a eInk-based tablet such as the Remarkable, which is Not Cheap (~500-600€) but seems to have very low lag (at least in the ads) and also doubles as an eReader and is capable of partial erasing; it is a very interesting concept, but the price tag led me to search more.
  • a Moleskine Smart Writing System, which uses actual pen and paper, except the paper is proprietary and while Somewhat Cheap (about 1€ for an A4) it is the only system that’ll require money for each page you use (wisely or less wisely).
  • tons of app&camera -based systems, some of which include microwave-erasable notebooks, such as the Rocket Book. These are cheaper (20-50€) but I didn’t like the idea of having to take a picture of the notebook, feeling in the end I’d just use it as, you know, a regular notebook.
  • tons of eWriter systems, which are Very Cheap (as low as 5€), with imperceptible input lag, but no “smart” features that allow exporting your sketches.

That’s where the Boogie Board Sync stood up. It is the only eWriter system with bluetooth capability and also has an affordable price tag (~60€). Due to the lack of reviews on the web, I’m writing my own.

Update 1: The Boogie Board Sync has been discontinued by the manufacturer, so I don’t recommend getting one.
Update 2: A couple more smart writing systems have surfaced, such as the Boogie Board Carbon Copy. I will talk again about those, along with other systems such as the Neo Smartpen, in another post.

The selling feature for me was the integration with Evernote. You can actually sketch something on the board, hit the save button, fire up the app on your mobile phone, let it sync, and have your sketch in Evernote moments after. They’ll all collect in a Evernote notebook of your choice for later reference. Unfortunately, Evernote is not able to recognize the hand-written text inside the sketches, but you can write some keywords (such as a timestamp and title) to make it easier to find it later.

The Boogie Board Sync app could definitely use some improvements. At first I tried to set it up on Windows to have it permanently available, but for some reason I couldn’t get the Evernote syncing to work on my laptop (tested on two Windows10 laptops). I tried to contact Boogie Board support but they never replied (boo!). This is really unprofessional on their part and brought me to the brink of returning the product, but in the end I figured out I could just install the app on my phone instead, which apparently works better. Also, I once had a problem where the app crashed, and upon restarting, it imported again every single sketch from the board, which had to be cleaned out manually; so even the app is not perfect.

The build quality for now seems ok. The board surface might look scratched from time to time, but it looks to be caused by the pen leaving some kind of small trail; if you clean up the screen with a cloth or a finger, you’ll have a perfectly smooth surface again.

The screen is still perfectly smooth after months of use.

On the other hand, it has shown to be vulnerable to hits. I’ve hit the screen probably by dropping my keychain on it, and now I have a couple of spots on the screen where it is much more sensible (for example, they light up if I softly push there with my finger).

A few hits to the screen led to some sensitive points. 

The pen stroke is somewhat thicker similar to the one of a soft felt tip pen, so you’ll have to adapt to writing with larger letters. Also, the screen sensitivity seems to depend on the heat (such as heat from sunlight) and it will grow bigger if the device gets hotter. Overall the vector version of your sketches is good enough to read later, but it is also not failure-proof, as some strokes (sometimes whole letters) will be missing from the end result. If your work involves symbols and numbers rather than words (such as, maths) I guess this could be a bigger issue.

How a note looks on the screen (left), and exported as PDF (right). You can notice some letters are completely missing.

A note on the software: when you delete a sketch, it is actually still accessible by using a USB cable and mounting the board as a USB thumb drive. The sketches are actually stored as vector PDFs on the board. The internal memory is probably enough for ~40k sketches.

The board can also be used as a digital drawing pad, it can left-click the screen by tapping the pen on the board, and right-click by using the button on the pen. It does not have pressure-sensitivity. This was not a relevant use case to me, but it still could be useful to someone.

Overall I’m rather happy with my Boogie Board Sync. I’ve averaged one note per day since when I had it 3 months ago and now I couldn’t go back. It definitely could use some improvements, but the concept is very interesting and as such I hope some competitors make their move with some new models.

Tablets comparison

Simone Leave a comment

As I digged through the available 8-10″ tablets starting from 200 and above, I compiled this list of the main features I was interested in. There doesn’t seem to be a lot of models.

For some reason, very few of them, if any, can do phone calls (which is bad, because I really hoped to use it as a backup in case my phone goes haywire).
LTE or 4G option costs about 100€.
Prices are in euros.
Also, see my other post about why a tablet isn’t a computer.

Model Price Screen size Screen type Resolution RAM Notes
Amazon Fire HD 8 110 8 IPS 1280 x 800 1,5
Apple iPad Air 2 Wi-Fi + Cellular 519 9.7 Retina 1536 x 2048 2
Not available on Amazon
Apple iPad Mini 4 409 7.9 Retina 2048 x 1536 2 Not available on Amazon
Asus Z300M-6B050A ZenPad 168 10 IPS 1280 x 800 2
Asus Zenpad 3S 317 9.7 IPS 1536 x 2048 4
Asus ZenPad 3S 10 LTE 419 9.7 IPS 2048 x 1536 4 No phone
Asus ZenPad 3S Z500M-1J006A 425 9.7 IPS 2048 x 1536 4 No phone
Asus Zenpad S 8.0 Z580C 289 8 IPS 2048 x 1536 2 No phone
Huawei MediaPad M3 324 8.4 IPS 2,560×1,600 4 Frontal speakers
Lenovo TAB 2 A10-70L 259 10.1 IPS 1900 x 1200 2
Lenovo TB2-X30L 175 10.1 IPS 1280 x 800 2 No phone
Lenovo Yoga Tab 3 Plus ZA1R0020DE 407 10.1 IPS 2560 x 1600 3
Lenovo ZA090058SE Yoga Tab 3 224 8 1280 x 800 2
Nexus 7 165 7 1920 x 1200 2 No phone
Nvidia SHIELD 16GB Black 333 8 1920 x 1200 2
NVIDIA Shield K1 200 8 1920 x 1200 2
Discontinued – no phone
Samsung Galaxy S3 632 9.7 Super AMOLED 2048 x 1536 4
Samsung Galaxy Tab A T585N 267 10.1 TFT 1920 x 1200 2
Samsung Galaxy Tab A6 194 10.1 IPS 1920 x 1200 2 No phone
Samsung Galaxy Tab E SM-T561N 202 9.7 TFT 1280 x 800 1,5 No phone
Samsung Galaxy Tab S2 T713N 329 8 Super AMOLED 2048 x 1536 3
Samsung Galaxy Tab S2 9.7″ SM-T810NZWEDBT 343 9.7 Super AMOLED 2048 x 1536 3 4:3 display
Samsung SM-T585 Galaxy Tab A 254 10.1 TFT 1.920 x 1.200 2
Sony Xperia Z 380 10.1 TFT 1920 x 1200 2 Waterproof
Sony Xperia Z4 Tablet SGP712 407 10.1 IPS 2560 x 1600 3 Waterproof
DELL Venue 8 Pro 205 8 IPS 1280 x 800 1 Windows

 

Some statistics about Italian names and surnames

Simone Leave a comment

I’ve had the chance to run some statistics on a list of Italian names and surnames. I plan to feed this list to a machine learning algorithm and see what I can find out, but prior to that, I was curious to search a few metrics about Italian names.

Italian names make up 78% of the names of residents, and I have restricted my searches to people with one of the first 1200 most common Italian names. With those two constraints, I have filtered out about 98% of all unique names, which is a huge percentage of names, but a comparatively small number of people: just 8% were left out.

As you can see there is a rather big push towards the most popular names:

 

21% of Italians also have a middle name, and the popularity of middle names is even more dramatically skewed.

So what are the most common names in Italy? Apparently, Maria and Giuseppe are the most popular by a long stretch. It probably isn’t a coincidence that those two names are important in Christianity.

MARIA 1,91%
GIUSEPPE 1,88%
ANTONIO 1,37%
FRANCESCO 1,22%
GIOVANNI 1,18%
MARCO 1,05%
ROBERTO 1,04%
ANNA 1,01%
MARIO 0,92%
LUIGI 0,90%
PAOLO 0,85%
ALESSANDRO 0,82%
ANDREA 0,80%
FRANCESCA 0,70%
STEFANO 0,70%
PAOLA 0,70%
VINCENZO 0,67%
LAURA 0,66%

While here are some of the most popular last names:

BIANCHI 0,24%
ROSSI 0,22%
FERRARI 0,18%
COLOMBO 0,15%
BRUNO 0,15%
GIORDANO 0,15%
ESPOSITO 0,15%
GALLO 0,14%
RUSSO 0,14%
PROIETTI 0,14%

Some of the most common combinations. As you can see some somewhat less common last names, such as Caruso or Marino, appear very frequently in combination with some first names.

RUSSO GIUSEPPE
BRUNO GIUSEPPE
RUSSO ANTONIO
MARINO GIUSEPPE
ROSSI MARIA
ESPOSITO ANTONIO
GIORDANO GIUSEPPE
ESPOSITO GIUSEPPE
GALLO GIUSEPPE
GIORDANO MARIA
FERRARI MARIA
CARUSO GIUSEPPE
BIANCHI MARIA
ROMANO GIUSEPPE
BIANCHI ROBERTO
MARINO MARIA
ESPOSITO MARIA
GIORDANO FRANCESCO
BIANCHI MARCO
ROSSI GIUSEPPE

Having a look at the average surname length, in number of characters. There are some interesting outliers here. The graph is cut at 22 characters, I’m not able to tell how far it could go. The longest ones are combinations of multiple surnames, joined with hyphens. If we look at the non-joined ones, we can find the longest are Silettiformantello and Pasquadibisceglie.

Also first names have pretty long ones, such as Francescantonio, Mariantonietta, Giovanbattista, or Domenicantonio.

There does not seem to be a correlation or relationship between length of first and last name; the line fitting the scatter plot is practically flat (m=-0.014).

A peek at the less common names (that still had their way in the 1200 most popular):

LOLA
EGISTO
FILIPPINA
CLARISSA
LIDIO
ALFIA
SALVINA
GILDO
DESOLINA
RINALDA
MASSIMILIANA
DIOMIRA
DENNIS
ARTEMIO
NARCISA
LEONTINA
LUCE
OLINDA
CLEONICE
BENVENUTA
MASSIMINA
EULALIA
RODRIGO
CRESCENZA
MARLENE
VITTO
CRESCENZIA

I’ll soon try to find out if I can use a neural network to deduce the phonetic rules that bind first and last names. First names should be carefully chosen, and it makes sense to suppose that it should exist a “rule” to decide if a first-last name pairing “feels” right or not. But this is stuff for another post!

Disclaimer: the database I used for this article is around 3.5 million names, mainly from bigger cities (Roma, Milano, Torino) and other (mostly northern) towns.

In 2018, a tablet still isn’t a computer

Simone 1 Comment

What is a computer? That’s the title of one of the latests commercials about the iPad Pro.

In a nutshell, the designers show that the tablet has nothing to lose when compared to a computer.

I beg to differ. I was recently searching for a replacement for a laptop, and the tablet was an interesting candidate; but after research and first-hand testing, I’m confident to say a tablet is not a computer. They are two different products for two different use cases. Sure, a 1000€ Microsoft Surface can get close enough to a laptop, and also the iPad has some niceties that make it above average in many areas, but things get worse if we look at the much more widespread Android tablet, which is the worst offender here, even for devices with a 600€ price tag.

There are many things a Android tablet cannot do well or at all.

  • You cannot have two apps running at the same time reliably, as Android likes to kill them randomly. This is especially problematic if you’re using tools such as IRC or SSH which happen to use connection-oriented protocols.
  • Physical keyboards aren’t remappable without rooting. For example the Escape key closes all applications and there is no way to disable it. (90% of keyboard shortcuts do work though.)
  • Support for external peripherals is almost non existing. While all Android devices have a USB port, this can’t be used for USB devices such as USB-to-ethernet adapters or USB-to-HDMI. So, no wired networks and no external monitors are allowed. If you need those, you’ll have to do some research, as support is extremely spotty (Samsung and Apple are among the best here).
  • Drawing is impossible as the latency is huge, especially on bigger screens. The only possibility here is to get a tablet that has support for an active pen (that currently is available only on the top tier ones). Also the pen itself is, again, expensive.
  • Filesystem support is cumbersome. Try to download a file in a new directory and then move it somewhere else. Or try to forward an email with an attachment on the iPhone.
  • Gaming on a tablet is a sub-par experience, as mobile games are not at the same level as mediocre PC indie games.

On the other hand, I don’t mean tablets are useless; they are fit for different purposes. Even many laptops have disadvantages with respect to a tablet:

  • Few laptops support SIM cards.
  • Laptop cameras aren’t as advanced as mobile ones, and a laptop anyway can’t easily take a picture of what it’s behind it.
  • Few laptops are touch-enabled.
  • “Flip” or “Transformer” designs are rare. The keyboard is a burden if you just want to watch YouTube. This basically makes a laptop much less portable.
  • Laptops just aren’t socially fit sometimes, you’ll look weird if you bring one at the beach.
  • Many mobile apps are designed around portability: maps, touristic informations, public transports, and so on. Unfortunately many of them aren’t available for all platforms; if you’re used to a certain app on your phone, you might need something different on your laptop.
  • Mobile apps often offer background services which are less resource-intensive than keeping tens of Chrome tabs open.

So, there are pro and cons on both sides, as they’re basically two different products with a slight overlap in functionality. Tablets still excel at entertainment, but laptops are still the most versatile and productive.
You should understand your use case before deciding for the one or the other. I hope this list helps!

 

Spaghetti Distance Index: measuring set similarity for sentences with a bag-of-words model (too literally)

Simone Leave a comment

Today we talk about how you shouldn’t measure sentence similarity.

“Bag of words” is a simplified model of handling sentences in natural language processing, which disregards grammar, punctuation, and word order and manages sentences as sets of words.

“Set similarity” is a measure of the distance between two sets (two collections of arbitrary items). There are many approaches in literature; I recently came across the Jaccard Index, while other alternatives are Sorensen Similarity Index and Hamming distance.
Jaccard Index (or Jaccard Distance) seems to be commonly used in text summarization algorithms, which use it to measure the similarity between two sentences (using a sentence as a set of words).

However Jaccard Index has some sub-optimal quirks. Its score is based on the number of items shared between the sets; when applied to sentence comparison, there are words that are very common (e.g. stopwords), especially in modern social network environments (e.g. Instagram hashtags), often used across different contexts (e.g. #picoftheday); I quickly realized not all words should be treated equally when measuring sentence similarity, and it is not trivial to evaluate if words are important or not.

Its score is also weighted on the set cardinality (the total number of words in the sentence); however this favours short sentences, for example it is way easier to have two sentences share all of their words when there is only one word in them. Big sets that have a large percentage of their items in common should get higher matching scores than smaller sets that have a similar percentage of matching items, since this has a smaller probability to happen for large sets.

This is where I tried to formulate a different, more context-aware, non-normalized, language-agnostic measure of similarity between sentences. I called this algorithm Spaghetti Distance Index. Its idea is, given two sentences, to measure the log-likelihood of ending up with a certain number of shared words. In formulas:

SDI(X,Y) = - \displaystyle\sum_{w \epsilon (X \cap Y)} log \left( |X| \ |Y| \ p(w) ^ 2 \right)

X and Y are the two sets, and |X| and |Y| are their size. p(w) is the probability of item w (simply put, the frequency of word w among all sentences). |X| p(w) is the probability of w appearing in set X, and similarly for Y. |X| \ |Y| \ p(w) ^ 2 is the probability an item w being in both sets X and Y. The less frequent is the item, the greater the probability of a relationship between the sets.

This algorithm has some advantages over Jaccard Index:

  • the similarity score is unbounded, so longer sentences can have higher scores;
  • it favours high percentages of matching words, non-shared words have a negative impact on score;
  • uncommon words contribute more to the score;
  • it maintains language agnosticity, and can be used to compare any set of items.

This algorithm assumes each word as chosen indipendently from the others; this is literally what a “bag of word” would get you if words are chosen blindly and randomly. On the other hand, this assumption is certainly not true: words in a sentence have a correlation between them. This is especially true for sentences belonging to a certain topic, i.e. if sentences are about pasta recipes, and one word is spaghetti, then there would be a somewhat high probability of another word being tomato.
Factorizing in the correlation between words would bring us to the Mutual Information algorithm, which relies exactly on the probability of a certain word given all the other words.

But if the problem is comparing sentence similarity, then possibly there are other domain-specific similarity measures such as tf-idf (which accounts for both term frequency in sentences, and for general term frequency among all words), BM25, or sentence clustering algorithms such as Latent Dirichlet allocation.

You can see the source code for Spaghetti Distance Index on GitHub.

POSIX Signals are not just bad, they’re awful

Simone Leave a comment

It all started with a problem which seem to be simple and already seen around in the real world:

I just need a simple script to restart a process if it crashes.

Simple enough right? So simple that I wanted to use a short bash script to do it. Turns out it’s not really that simple, especially if your thought process continues with

And I will use signals to control it.

Okay, now it’s really a bad idea which can fail for a lot of reasons:

 1. Checking if a process has crashed is not idiot-proof

The idiot being the one that writes a C program that exits main() with “return(-1);” or any other negative value. The exit status is a signed byte, thus a negative value will wrap around to a > 127 value and be indistinguishable from a process crash.

A program exiting with “return(-119);” has the same exit status as one killed with kill -9 (SIGKILL). The fun!

 2. You can’t always control signals in bash

According to POSIX specs, bash can’t change the signal handler for signals which were ignored in the parent process. So this script would be unusable if you happen to launch it from a process which has the relevant signals ignored. That’s where I got the idea to use a one-line python wrapper around my bash script to enable all signals before giving control to bash.

 3. Using a signal wrapper spawns a second process

It turned out quickly that it wasn’t a great idea. Using a python wrapper around a bash script will obviously generate a second process, one for bash and one for python.
So in just one move I spawned three more problems:

– the python wrapper has the process name you would expect (the wrapper name) and has every signal enabled, i.e. any signal would kill it
– the bash script has a different process name which makes it counter-intuitive which one is the process to send signals to
– the bash script starts after the python wrapper so if two wrappers start simultaneously, we have a funny race condition to deal with.

So, I just decided to drop bash and rewrite everything in python instead.

 4. There is no standard signal to ask a process to restart

Usually it’s SIGHUP, but it’s not universally true. If your controlled process can be restarted with a signal, that signal should be SIGHUP, but no guarantees.

 5. Signals are not setup immediately at startup

When your control script starts, all signals have their default handler. So for example if you launch your control script and then immediately decide to restart the controlled process (with a SIGHUP signal), it may happen that the controller gets killed instead, and the child process is left with no control.

 6. SIGKILL can’t be handled

If your control script is killed with SIGKILL, the child process is left running with no control.

 7. The child process exiting will have interesting race conditions

Say for example that the restart behaviour will be to send SIGHUP to the child process, wait for its exit, and launch the child process again.
If your control script is asked to restart the child process several times in a row, it might be sending several signals to a PID, which after the first SIGHUP might not correspond to any running process (not so bad) or correspond to a different process recently spawned (definitely not good).

In certain *NIX flavours you can setup a signal handler that will fire when the child process exits, but based on my research this is not true for every OS out there.

 8. There is no alternative to signals

I might have finished this list with better news, but unfortunately the only way to ask a process to terminate is, yes, signals.

The DOs:

 Use a lockfile to ensure a single instance of your controller is running.

 Use process groups.

 Handle SIGCHLD if your OS uses it to signal child processes exiting.

 Use systemd if your OS has it. It might get kind of long to config properly (especially for a dynamic list of processes), though.

PHP not parsing POST data?

Simone 1 Comment

If your PHP (or any other server side code) is receiving a single $_POST field of raw data, looking more or less like this, instead of receiving your form fields nicely:

——WebKitFormBoundaryVmWPyJJIBR3n18Wk
Content-Disposition: form-data; name=”form-file-1″; filename=”my document.doc”
Content-Type: application/octet-stream
——WebKitFormBoundaryVmWPyJJIBR3n18Wk
Content-Disposition: form-data; name=”form-field-1″

form-value-1
——WebKitFormBoundaryVmWPyJJIBR3n18Wk–

(that WebKitFormBoundary is probably the sign that you are using Chrome)

Then, chances are your javascript is overriding the content-type header. Check out the request headers, and if you find anything looking like this:

Content-Type:application/x-www-form-urlencoded;

or really anything else which is not this:

Content-Type:multipart/form-data; boundary=—-WebKitFormBoundaryVmWPyJJIBR3n18Wk

then you need to fix the content-type header. It might be simpler than you think; try setting it to `false` (in jQuery / Prototype / …) or disable anything javascript is setting there. The browser will take care of it better than what JS is doing.

In my case, I was getting this problem because I was using the js FormData object to extract form data and then sending it the same way it used to do with “regular” form data. Turns out the content-type header should be set differently. Maybe I saved you a couple of hours of searching.

How to not drive a motor with an Arduino

Simone 2 Comments

If you are like me, when you were an absolute beginner, you tried to wire a motor up to some batteries, saw the motor was spinning, and thought out loud: “It’s working! How hard could it be to turn it on and off with an Arduino?”

Actually a little more than it seemed at first. There are several problems I encountered and slowly fixed, and I’ll try to sum up in this post the kind of iterative reasoning that led to a decent circuit design.

motor1

So, first thing you try is to attach the battery poles to the motor, keeping the wires in place with your fingers. Everything seems to work fine! So what’s next? What’s the most obvious attempt to control it?

motor2

The second step would be to add a switch. Pushing the switch makes it run, release the switch and it stops. Perfect: now you just have to use an Arduino to open and close that switch.
motor3
This might be the most obvious change for a beginner, but also a very very bad idea. All the current is passing through the Arduino which is, in best cases, limited at 40mA, while even a very small motor will probably need way more than that. This might work for very very small motors but overall you risk damaging the pins.

motor4

So a “better” (but still bad) idea is to use a transistor instead! The configuration above is called emitter-follower. After all, transistor are like switches, and you have a lot of 2n2222 lying around, and they are a buck a kilogram, so why not make use of them? And maybe this circuit is kind of working, but it has at least two issues you should be considering now:

  • What is the current required by the motor? Check the datasheet, and remember it probably needs a way higher amount when it is starting from a still position. The 2n2222 has a maximum rating of 500mA which might not be enough even if your motor is rated to consume only, say, 100mA. A cheap option would be to use a Darlington pair (basically, more transistors), but a better choice would be to use a different transistor which can bear a bigger current, like a MOSFET (e.g. a IRF510, still cheap and easily available) or possibly a relay. (As you probably know there are several models of BJTs, MOSFETs and relays and I’m pretty sure someone skilled enough could write a book about proper choice of this part).
  • the other issue is that a part of the current is still sourced through the Arduino, so before burning something for good, a nice idea would be to add a resistor for protection, so that the maximum current flowing out of the Arduino is limited. (This is probably not an issue if using a MOSFET).

motor5

This will add some protection to the Arduino, but really, why risk? We can change the circuit so that none of the current flowing from the Arduino actually goes to the motor.

motor6

This configuration, called common emitter, is getting better, but there’s still something needed to be done. The motor is a typical inductive load, and when the transistor tries to open the circuit, the motor will try to keep going, forcing a current in the transistor (which is bad and will likely damage it) and/or in the Arduino (which will cause random resets and damage the pins). Another improvement is to add a flyback diode, so that the current has somewhere to go when the circuit is open.

motor7

There. It probably isn’t finished here, but it’s getting on the safe side. Then you might want to drive the motor in reverse, vary its speed, but this will probably be discussed in another post.

(Credits: graphics in this page were drawn with Fritzing.)