More yspell weird: extra spaces after double quotes

So we continue to use Yahoo Spell Suggest API for spell suggestions in our catalog and article search interface.

I previously reported on some weird undocumented issues with percent-encoding in Yahoo spell responses.

Today, I fixed another weird issue.  For any query that included a double-quote (“) in it, the yahoo spell suggest would make a suggestion to put a space after the double quote. What?

So if you searched:

"Noam Chomsky"

The spell suggestion would be, Did you mean:

" Noam Chomsky"

Well, no, I didn’t. And in our search interfaces, the extra space doesn’t actually do anything different (if it did, it would probably be something undesired, but I don’t think it does).

I first noticed this about a month ago, but am only getting around to fixing it now. Has Yahoo Spell suggest been doing this all along?  I don’t really know — I feel like I would have tested phrase searching when I first implemented last March, and noticed this issue then. And a librarian user also only reported this problem to me recently. But I don’t know, maybe it’s been doing it all along and we hadn’t noticed; or maybe Yahoo spell suggest actually gets regular updates with changed behavior from Yahoo.

Regardless, it was annoying and confusing users, so we put in a workaround. My code that gets and supplies the suggestion from Yahoo now erases any spaces right after double quotes in the suggestion; and then makes sure the normalized suggestion is actually different from the regular query before offering it.

suggestion = suggestion.gsub(/\" +/, '"')

Now if you

  • search for «”noam chomsky”», yspell will suggest «” noam chomsky”», but we’ll ignore it.
  • search for «”noam chmsky”», yspell will suggest «” noam chomsky”», but we’ll normalize to «”noam chomsky”» before we offer the suggestion.
  • actually enter a space and search for «” noam chomsky”» — I worried that we’d end up suggesting «did you mean “noam chomsky”» to you, which might be annoying, but at least at the moment, entering a space after a double quote yourself like does not seem to result in any suggestions from yspell at all.
  • If you actually enter «” noam chmsky”», ypell will return «” noam chomsky”», and we’ll normalize the space out to «”noam chomsky”» before we show you the suggestion, which I think is probably okay.

The number of known idiosyncracies and weirdities requiring workarounds from Yahoo Spell continues to increase; nevertheless, I don’t know of anything else available that works better at the relatively affordable price point of Yahoo Spell Suggest. 

I still wish it didn’t have the oddities and undocumented weirdnesses it does have.

This entry was posted in General. Bookmark the permalink.

3 Responses to More yspell weird: extra spaces after double quotes

  1. Robin Sinn says:

    Thank you! I’d been meaning to ask about this, but was always distracted by something shiny. It’s been going on for maybe a month?

  2. jrochkind says:

    Not really sure how long it’s been going on. You hadn’t noticed it until the last month or so, like me? Our workaround is in place now.

  3. jrochkind says:

    Oy, I also just noticed it’s sometimes making suggestions where the only difference is a case difference too. That’s harder to work around, we’ll see how much of a problem/distraction it is for users. Yahoo spell api, definitely not perfect, but defintiely seems to be appreciated by users anyway, I think — and I don’t know of any better options.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s