Today we’re excited to unveil an enhancement that makes using Highrise even more convenient: The sidebar search-for-a-person feature is now significantly faster than before.
Searching for a person or company from the sidebar on the Dashboard or a person/company’s page is the most frequently used feature in all of Highrise. Highrise is about getting to a person/company’s page so you can enter a note or look up a previous conversation or grab a phone number. Now you can do that a whole lot faster. More speed and less wait time makes this experience markedly better.
Making this faster on the user experience side wasn’t the only goal here: The new sidebar search also reduces call-backs to the server. That lowers the number of requests to the database which, indirectly, makes everything else a little bit faster too.
Watch this video to see it in action
Sam Stephenson, one of our developers here at 37signals, has been working hard to make this a reality. And now that we’ve launched it, he put together a video showing you the before and after:
We hope this helps makes using Highrise an even better experience. Thanks for your continued support!
Dhrumil
on 30 Oct 08This is fantastic.
Mimo
on 30 Oct 08Sleek.
Joost Diepenmaat
on 30 Oct 08Looks great! Any background information on the search techniques Sam used to implement this? I’m really interested in the search mechanism behind this.
Joaquin Bravo
on 30 Oct 08So… how did you do it? =)
Derek
on 30 Oct 08Ditto on the how did you do it… (Looks like you got some JS caching happening to achieve that speed)
But what I’m really interested in is how did you get a result for joshstw to point to Josh Stewart??? That looks like voodoo to me!
Bryce
on 30 Oct 08@Derek: matching for anything that contains ‘joshstw’ in that order but without caring about intermediate characters.
Robin Hood
on 30 Oct 08Very cool. Works great if you have the full name in a field by itself, but what if you wanted to search multiple fields of a contact record?
Say Josh Peek works for the company 37 signals. If you type in 37 signals, then it would be great to show you all the contacts from that company. This is when you start to use things likie Sphinx/Lucene over normal text searches.
Henrik N
on 30 Oct 08The placeholder text is nice. Kind of hypnotic. What library do you use for that?
Not so sure about “superanal” (in the task list) though…
Henrik N
on 30 Oct 08If this is just your contacts, not a huge set of global users, I would guess they simply put the list into a JS array or similar when the page is generated, so there are no extra HTTP requests.
Mike Rundle
on 30 Oct 08Looks like you guys just loaded an array of all accessible names into the page on load, then are iterating through that on each character press. Not that fancy, but techniques like this make a really big difference.
blackant
on 30 Oct 08No, it’s not loading all of the contacts into the page (and I would hope not in Sam’s demo with 16,000 of them). A quick test with Firebug shows Ajax requests going out when you type.
I’d also be interested in learning about the server-side magic – this is really fast and I like the TextMate-ish friendliness of being able to omit characters.
Great work guys!
Dmitry Chestnykh
on 30 Oct 08Nice!
When you add a new person after search, wouldn’t it be better to focus on the title field instead of the first name field (because it’s been pre-filled)?
JF
on 30 Oct 08When you add a new person after search, wouldn’t it be better to focus on the title field instead of the first name field (because it’s been pre-filled)?
Yup, we agree, but it was out of scope for this initial release.
David Reese
on 30 Oct 08Just adding my name to the chorus of people asking about implementation—I’d be lots of people would like to do a similar thing with Rails and solr/sphinx/ferret. The specific question I’m dealing with is scoping out the fulltext search per subdomain/account…
if you’d like to share, of course!
John Kranz
on 30 Oct 08Very nice enhancement. At the end of your demo, when you add a new person, wouldn’t it be best to have the text cursor positioned in the title field since the first and last name have already been entered in the people search box along with confirmation you want to add that new person?
John Kranz
on 30 Oct 08Sorry, seems my post was being written before the comments above it appeared….
Justin Reese
on 30 Oct 08That’s some incredible performance. Out of curiosity, was Sam doing the demo from a local Highrise installation, or does that represent true real world performance (with network latency)? I don’t have access to a high-contact-count Highrist account to test for myself.
Paul Smith
on 30 Oct 08Works pretty nice! It’s fast, but there are some bugs that are a little bit annoying. One thing is the dropdown doesn’t go away unless you delete the text. So it covers the add task and add person buttons. Also the onresize event on the window should reposition the drop down. Not a big deal, but just letting you know. It shouldn’t be too hard to fix the onresize, or the onblur for the search field.
Oh and I would leave focus on the first name field in case it screws up the name. If you type “joshstw” it will fill in the first name as “Joshst” so you’d need to fix it. Either way it isn’t a big deal though. They both would work well.
Daniel
on 30 Oct 08That really is awesome performance! I was a bit skeptical seeing it run on the test server, but I just tried it here from Scandinavia, and.. dayum!
My guess (haven’t bothered to check Firebug) is that all new results pertaining to a query are cache’d in JS. So provided you keep adding letters to the same basic query (e.g. “jo”, “joh”, “john”, it’ll whittle down the initial result set in JS. If you go and change the first couple of letters (e.g. “john” to “ja”, “jan”, “jane”) it’ll make a new database query for “ja” and whittle away in JS as you type. Maybe. As I say, it’s just a guess and I haven’t checked, but I’d probably try that myself, if I was building it.
But that still doesn’t answer the question of how the database searching has been sped up so much. I’m in total awe of the speed of that. Are there any limitations on the number of results? For instance if you type “aa” you’d probably find thousands of people in the 16k test database – but that’s a bit too much to send and to use, I would think. Also, if there are any result limits, what are the criteria (if any) for omitting or including results? I don’t have a large enough Highrise database to hit any result limits.
Paul Smith
on 30 Oct 08@Daniel: I think even if it return 1-200 results it would still be reasonable. I did a couple queries and an it returns on average 20 results per KB un-gzipped. So 100 queries would only be 10 kb, and even less when gzipped. I bet it just returns them all.
SS
on 30 Oct 08Some technical details for those who are interested:
Searching begins as soon as you type a second non-space character in the search field. Your browser makes a request to the Highrise servers, and Highrise sends back the names and URLs of all people and companies visible to you whose names contain sequentially the two characters you typed. (For example, if the first two characters you type are “ja”, Highrise will send back “Jason Fried” as well as “Jim Coudal” and “Sanji Fernando.”) Typing more characters sorts and pares down the result set in your browser without hitting the Highrise servers. Highrise displays at most 50 results in the list.
Avoiding multiple search requests was the first strategy for improving speed. On the server side, we’re doing a few more things to make that search request fast. In order to efficiently query the database for partial matches spanning full names, we denormalized the first- and last-name columns in the parties table (which holds every person and company) into a third full-name column. Then the query is just a LIKE clause against the full-name column (e.g. full_name LIKE '%j%a%') scoped by account and user permissions.
Side story: During development we had the full-name column directly on the parties table, but when we ran the migration on our staging servers against a copy of the production database we found going with that approach would require over two hours of downtime. So we reworked it to store the full name in a separate table and avoided downtime entirely.
Another big performance win was instructing Active Record not to instantiate Person and Company objects for each result, but to return a simple attributes hash for each record instead. This makes a significant difference in large accounts where filtering by only two characters can return hundreds of results. The query itself may be fast, but creating hundreds of new AR objects (each instantiation in turn creating hundreds more Ruby objects) is not.
Finally we needed to make sure that once the results were retrieved they got into your browser as fast as possible. There are two ways to quickly load large chunks of data into a web page: setting a DOM element’s
innerHTMLproperty, or usingevalto load JavaScript objects. Normally we prefer the former approach whenever possible, but the latter is a much better fit for this particular feature. So the search results are sent from our servers as a JSON array, and we gzip that response so we can transfer as little data as possible.SS
on 30 Oct 08By the way we now focus the first blank field (either last name or title) on the new person form when you choose “Add a new person with this name.” Thanks for the suggestion.
David Reese
on 30 Oct 08Thanks a lot Sam, that’s a brilliant implementation. (Wouldn’t expect any less, of course).
Never would have imagined you’d just do a simple LIKE ‘xx’ query with your quantities of data… I’ve only got 150k records and I’m knee deep in sphinx. Gotta rethink.
Daniel
on 31 Oct 08Yay! I guessed right! :) Do I get a prize? I do so like prizes
I must admit though, that I hadn’t figured you’d do a simple LIKE query either. And I completely ignored the browser rendering issue. I was just thinking, that I’d probably have gone with innerHTML, and not thought more about it (darn you Sam for Ajax.Updater – makes it too easy!), but realized that since the result set needs to be cached and remain searchable in JS, JSON is of course the way to go.
Very instructive example with a very elegant solution and a tremendous speed boost; Congrats, guys! Excellent work as always.
MI
on 31 Oct 08David and Daniel, don’t feel bad, I couldn’t imagine that a simple LIKE query would have been sufficient either. I argued quite vehemently against that design for contact search when we initially implemented it in Highrise a year and a half ago—and I was wrong. Even with millions of contacts in the database, it turns out that the working set for a particular account and user is sufficiently winnowed down that the query is reasonably fast.
Just goes to show, don’t trust your instincts where performance is concerned. ALWAYS benchmark.
Derek
on 31 Oct 08@SS Thanks a lot for the explanation of this, I’m now just a little smarter than I was before. I use a lot of Like statements in my work but I never though of using it that way.
I see why you might want to use eval for the building of the array, but considering the security issues around eval (not really applicable to this case) is it really that much slower to just send a JSON object as gzipped text to you JS and have it parse it into an array? I’m going to have to do some benchmarking to see how that works.
Thanks again for explaining it in detail for us web heads.
Daniel
on 31 Oct 08@MI Heck, I’m happy to be surprised! And truth be told, I’m a complete amateur: I don’t manage a large-scale web app, though I do do some small scale web development and design.
Last time I wrote a search function - for a web shop with only about 3k products and categories - I used LIKE as well, and figured it’d probably be slower than other techniques. It turned out to be plenty fast, but then it wasn’t dynamic but a bog-standard GET page request. Didn’t benchmark it against anything, I just tried it and it was plenty fast, but I figured it was because the database was miniscule.
Fortunately for me, wasting time investigating and implementing more advanced queries was out of the question, since this shop runs on dirt cheap (but quite ok) shared hosting with garden variety PHP and MySQL being the only tools at my disposal. I’d like to use Rails, but none of the projects I’ve done justify the price of Rails hosting. I just wrote my own PHP framework instead :)
Like you guys say: Embrace constraints :)
Oh, one last anecdote: The framework I mentioned is basically a rip-off of Rails (sorry, guys) because I craved ActiveRecord and the Model-View-Controller goodness (wound up with a lot of other stuff too). But with the search function I too found that instantiating record objects for the results was no-go. Not because of speed, but because PHP ran out of memory on the greediest searches… Ah, the wonders of cheap-o hosting.
Paginating results of course solved it, and had always been the plan anyway, but it’s just to say that speed is usually a purely theoretical concern for me :)
StartBreakingFree.com
on 31 Oct 08Pretty elite coding guys…thanks for sharing the technical details.
One point of clarification, when you do the “full_name LIKE ‘j%a” does it return results first where the ja are together, or do you have to sort it some way?
Great stuff! Now just need to make it a plugin :) Brian
Dan
on 02 Nov 08The Evolution of Auto Complete http://www.logblo.com/2008/11/02/TheEvolutionOfAutoComplete.aspx
Russell
on 03 Nov 08Did you mean to share your freelancer rates?
Robin Hood
on 05 Nov 08Thanks for the technical details.
The LIKE clause works great when you are able to search one text field and not include a weighting on parts of that field. Other tools I’ve used enable you to type anything in the search box and it pulls up results based on multiple fields.
A book for example has a title, author, ISBN, publisher, etc… I find it to be more helpful to have one area to search everything.
Say the phone rings at my office. I don’t recognize the number and don’t have it stored in my phone. I’d love to type that number in my Highrise search box to see who is calling me while answering. This is when you would either have multiple conditional LIKE clauses or move to a sphinx/lucene engine.
Is this ‘global search’ a feature requested a lot by your customers?
This discussion is closed.