We’ve been working for some time with python-calais, a Python wrapper for the Calais Semantic Text Annotation Service led by Jordan Dimov.  For Likematter, we extended its native support for Calais’ JSON response type to include support for Calais’ RDF output.  Several people expressed interest in using rdflib with Calais at last night’s Python meetup in Cambridge, MA - so I’ve packaged up our RDF extension as a starting point/example for folks looking to process Calais RDF in Python apps.

The reason to use RDF over JSON is that the RDF output contains a fuller representation of the analysis Calais produces.   As just one minor example, the full de-referenceable URLs for each entity type (e.g. Country, Person, etc.) aren’t present in the JSON.  The drawback to using RDF is that it’s much harder to interpret and translate into Python objects than JSON.   The code here includes the SPARQL to obtain only part of what Calais produces, entities and categories, but should serve as a starting point to working with Calais’ RDF.

Our RDF extension has been  added to the python-calais google-code repository.  The key SPARQL queries for categories and entities are below.

CATEGORY_QUERY = { ‘fields’ : ['docId', 'category', 'categoryName', 'score'],
‘SPARQL’ : “”"
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX cp: <http://s.opencalais.com/1/pred/>
SELECT ?docId ?category ?categoryName ?score
WHERE { ?doc cp:docId ?docId .
?doc cp:category ?category .
?doc cp:categoryName ?categoryName .
?doc cp:score ?score . }
“”"  }

ENTITY_QUERY = { ‘fields’ : ['entityId', 'name', 'type', 'relevance', 'resolves_to_uri', 'resolves_to_type', 'resolves_to_name', 'resolves_to_score'],
‘SPARQL’ : “”"
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX cp: <http://s.opencalais.com/1/pred/>
SELECT ?entity ?name ?type ?relevance ?res_uri ?res_type ?res_name ?res_score
WHERE {?entity cp:name ?name .
?entity rdf:type ?type .
?rel_uri cp:subject ?entity .
?rel_uri cp:relevance ?relevance .
OPTIONAL { ?res_uri cp:subject ?entity .
?res_uri rdf:type ?res_type .
?res_uri cp:name ?res_name . }
}
“”"  }

It’s intuitive that reducing information, much like eliminating functionality, simplifies our tasks, making it easier and faster.  But is the resulting decision better?  Do we achieve better results with more data to work with, more capabilities in our tools?  Gigerenzer’s research is about showing that not only does less mean easier, but less actually means better results.  He cites evidence from his work and elsewhere to support his claim that simple rule-of-thumb heuristics can actually outperform data-hungry, expensive algorithms in making a wide range of decisions.

Making sense of massive amounts of data in real time with limited computing resources requires the same problem-solving process that he describes in this book:

1. Get good at knowing what information is important
2. Have simple rules, but make sure that they’re the right ones

What might seem like a shortcut can actually produce a user advantage on top of being cheaper and faster.  This is encouraging!

I’m curious to know if anyone has looked at this effect with respect to user experience.  What, beyond eliminating negatives through ease-of-use, speed, efficiency, and cost savings, has the constraining of Twitter’s message size or Google’s search page layout done to create the core value that’s made these huge successes?

1. I want more understanding and control of how new content is matched to the things I like.

The most noticeable aspect of the new interface is the color-highlighting of content based on it’s relevance to my interest profile - thanks to Aaron VanDerlip of Jazkarta for suggesting this enhancement.  Numeric scores are also displayed with a red score indicator - clicking on the score displays detail about the source and specific concepts that factored in to the score.  These improvements will make it easier to understand how content is prioritized for each user.

User also have more control over how the system works with this new interface.  As shown in the screenshot, the like button now offers a preview/edit window.  By default, the source and all key concepts are added to your profile.  You can edit these selections and tag the article with concepts we failed to capture automatically.  And each user’s profile now displays an overview of the collection and a tag cloud tracking the concept present in the material you’ve collected (see screenshot).  Concepts are sized to estimate the strength and uniqueness to this individual user, using a calculation based on term frequency-inverse document frequency(tf-idf) weighting.  In the near future, users will be able to use this cloud to modify their profiles, control how the system is working for them, and manage their collections to easily refer back to items they’ve bookmarked in the past.

2. As a new user, the experience is frustrating because I’m not seeing articles that I like fast enough - is this thing working?

In the new release, we’ve increased the sensitivity of the matching algorithm so that results should appear immediately, and we’ll continue to make improvements in how well it performs with profiles that have only a few liked items.  The engine works better as profiles grow, sometimes requiring 10 or more items to reach a steady level.  To overcome this, we’re working on two new features that will make it easier to build up a new profile.

Coming soon:

• Search Box - A search box, which can be used to focus on specific topics and populate your profile with much wider range of concepts
• Import Favorite Content - bring in your bookmarks and more from other sites you use to collect stuff you like

I’m pleased to announce that the Likematter alpha is now available for open registration.  Likematter is a personal lens for the web.  It helps you stay on top of emerging content like breaking news and career opportunities.  In doing so, you automatically build a profile of your interests that you control and that will get better and better at filtering the latest information for you.

With some good lessons learned from our limited-availability pilot, we’ve made big adjustments.  Most significantly, we’ve added breaking news to the personalized job listings that were the focus of the original JobGravy career site.  The most resounding feedback we got from a group of much-appreciated early users was that they should be able to apply their interest profile to content beyond job listings, especially breaking news, and that the process of building the profile should be easier.  The result is re-engineered service, called Likematter, that hits both of these points:  1)Your profile can be used to filter news and jobs today, with more content channels coming soon.  2)The main way to indicate what you like is by using the Like it! button highlighted in red in the screenshot above.  There will once again be support for import of bookmarks and favorites from other sources soon.

Likematter is easy to use:

1. Sign Up
   
2. Start reading - start reading news or browsing jobs (jobs will be back online shortly). Clicking an entry expands it to reveal additional information.
3. Like what you like - Click the  Like it! button when something interests you.
4. Likematter learns - the system learns from your choices over time, prioritizing your topics so that your unique interests are emphasized.
5. Find what you want automatically - Likematter gets better and better at displaying content that relates to your interests as they evolve.  The red bars at the right of each line (see screenshot) indicate the strength of the match.  They make it easier to scan the page for good content.
6. Maintain control - Likematter works well without any additional tweaking, but with the forthcoming release of profile management, you’ll be able to refer back to content you’ve liked in the past, adjust how your profile filters content, and have greater control over the content sources and publications you have access to.

What does it do for you?

1. Helps you stay on top of real-time information across a wide range of interests, automatically - As your profile grows, Likematter gets better at making sure the content on your page is the best, most current information across the whole range of diverse interests you have - without having to manually tell the system what you want.  So, in addition to being able to search for specific topics (search bar coming soon, along with controls like paging and a time window zoom control to point Likematter at the right ), you are able to sit back and let Likematter search for a range of things you’re likely to find interesting and put them in front of you in a format that’s easy to read and navigate.
2. Develops valuable information about your personal interests - data which you own and control - Likematter also functions as a way to very easily bookmark and record what you’re interested in over time.  Eliminating the need for tagging, Likematter automatically extracts the key concepts from each item you like, using the powerful Calais Semantic Web Engine.  As the profile management capability currently under development becomes available, you’ll be able to refer back to the material you’ve saved, organize your information by topic, control how Likematter works on your behalf, and share your interests and the content your profile generates with friends.  You have complete control over your profile because your data belongs to you.  We commit to allow you to easily export your data at any time and agree to remove your data from Likematter at at your request - see full data policy for more information.
3. Links my news interests to career opportunities - By using Likematter to find and read the latest news that you like, you develop a profile that carries over to help you in other areas like staying aware of good career opportunities in your field.  Switch over to the the Jobs tab and the interests built up from reading news will automatically apply to identifying good job listings.  In the future you’ll be able to use your profile to help you find good content in areas of your choice by pointing Likematter at real-time media like Twitter.

The development of Likematter is led by Denise Ichinco and Mark Soper (personal, work, LinkedIn, Twitter).  We welcome thoughts, questions, and other feedback.  Enjoy!

Mark

Welcome to the new JobGravy Blog!

JobGravy is a new internet career service startup based in part on the technology and business efforts of Alluvial Labs, which (along with the original Alluvial Labs Blog) will be a part of JobGravy moving forward.  We are working on technology that will allow people to organize their own internet data footprints into structured information assets that they can use in a career context.  For example, you could with little effort use it to monitor all new job openings that fit at least five of your more unique (less widespread in job market) interests without having to know which five in advance and without having to fill out any resume profiles or questionnaires.  We believe this kind of capability will mean those doing targeted searches can find better leads more quickly and will allow the folks who might otherwise not be willing to invest in staying alert for the potential lifechanger to overcome the hurdles to do so.

The service is in development, with an alpha program being rolled out in the near future.  I welcome you to check out the information about the company and product on JobGravy.com and look forward to hearing your thoughts about it.

Mark Soper

(*) A special thanks to Vera Novakova for designing the JobGravy logo

Paul alludes to comments made by Gautam Godhwani of Simply Hired.  Gautam pointed out that the reason search engines like Google occupy such an important commercial position is because search is such a good indicator of a person’s intent to take immediate action.

Kevin Krim of Yahoo! Hot Jobs later described much room for improvement in techniques used in job search and other job matching applications, suggesting that we can do better at using available information to make the process easier for employers and professionals.

JobGravy answers Kevin’s call for better matching by expanding Gautam’s notion of intent to include a broader set of information about the individual’s interests (i.e. history of intent) in the search process.

JobGravy is under development, with an alpha release to become available in October 2008.

Top 25 Technologies, Programming Languages, Operating Systems, and Medical Treatments occurring in 5,000 tech job postings (*)

The following statistics can be used to approximate the relative popularity of various computer and medical technologies among employers in August 2008, with the caveat that the sampling was not done rigorously and is not necessarily representative of the broader US job picture.

(*) Results based on a sample of 5000 job listings from major, tech-oriented job boards in August 2008.

• Matthew Hurst provided interesting big-picture perspective on the current and future states of social media analysis based on his research experience at Microsoft Live Labs.  I really appreciated his breakdown of the present state into two realms:
  • Content - techniques that analyze the content of a document (e.g. entities, sentiment, topics, etc.)
  • Structure - techniques that analyze the network context of a document (e.g. links, threads, etc.)

  He then described an overall process built around these two core pieces, including mention of a third realm of temporal analysis - it would be nice to get more explanation on this and on the whole present state picture as Matthew didn’t have a chance to go into much detail.  I really liked that he took a complex situation and explained it in straightforward way - one of those moments when someone describes something, about which you’re already familiar, and without introducing a lot of new information, helps you to gain valuable insight.

  His response to my question about where tagging fit into his model was insightful, though I don’t think I’m yet fully convinced about how applicable the first statement is to many applications of tagging.  He said that they have generally found tags to be of little value when applied to documents because the information in the tags is often redundant with respect to the information in the text.  He did mention that that tags are highly valuable when applied to other media types like images (e.g. Flickr’s very successful tagging system) - right on the money.

  Matthew was good enough to post his slides along with a recap.

• Though not related to social media specifically, definitely deserving mention was Justin Langseth’s 1-hour demo of a Clarabridge implementation.  Not to be confused with an end-user demo, this was a live walk-through of the entire process of deploying and then using their text analytics platform against a modest customer feedback dataset, in one hour.  Clarabridge technology, and Justin’s technical presentation of it, are pretty impressive.  These guys understand BI, and seem to have figured out a good way to structure text data such that it works well within the major enterprise BI frameworks, along with the necessary infrastructure and admin tools to hook everything up without a lot of pain.  In my opinion, their current Windows orientation is less than desirable from a technology perspective.  But their platform decisions, like the use of J2EE, suggest that they appreciate the importance of portability long-term.  And if the enthused customers in attendance are any indication, the business value of what they’re doing far trumps this concern.
• Andrew Bernstein, of Cymfony, related insight from his work in helping companies with large consumer brands monitor market sentiment, test-market new products, and implement market intelligence using data sourced from MySpace, Facebook, blogs, etc.  Interestingly, his four-quadrant social media matrix recommends that companies focus their analytics efforts on social networks first with blogs a close second in terms of market intelligence value.  Social bookmarking, perhaps not surprisingly, faired poorly on this list.  Much more surprisingly given its heavy overlap with both blogging and social networking, Twitter ranked in last place as a valuable source of market intelligence.

Note: The following account of BarCamp Boston benefits from the additional time for reflection and inevitable blurring of facts afforded by the week’s vacation that immediately followed it.

BarCamp Boston is an annual gathering of Boston-area technology people, part of the BarCamp series.  This year’s meeting occurred over the weekend of May 17-18th in Cambridge, MA.  It turned out to be an excellent venue for developing entrepreneurial ideas related to web technology, social networking, mobile applications, software development, etc:

Sessions self-organize on the fly, allowing attendees to promote topics, receive feedback, and schedule talks in real time.  The result is that you have good options at any time to find an interesting topic or create one and lead a discussion around it.  This format also seems to have encouraged some experimental, and especially cross-subject discussions (e.g. At the Intersection of Everything, and the data-visualization discussion held by Matt McKeon of IBM’s Many Eyes).  A discussion I proposed on Natural Language Processing & Semantic Web API’s, covering Calais, Textwise Semantic Hacker (& the $1M Challenge), Y!Q, etc. produced some insight on what to expect (an not expect) from semantic technologies, and on suggestions from web developers on what will be required for more widespread adoption of RDF and other technologies for putting content into more machine-processable context.  One major theme: Better top-down tools are needed to make the process of adding context more automatic.  This ReadWriteWeb article contains a good overview of semantic technologies covered in the discussion.

Warm thanks to Shimon and Mike Walsh for making it happen.

Mark Soper

This morning’s Social Media Breakfast in at Ryles Jazz Club in Cambridge included a speaker’s panel focused on hiring and getting hired using social media. The speakers included:

• Stever Robbins, consultant/executive coach and host of the Get-It-Done Guy Podcast
• Aaron Strout, VP of new media at Mzinga.
• Todd Defren, principal at SHIFT Communications
• Chris Brogan, VP of strategy & technology at CrossTech Media

They touched on two main themes:

1. Using social media to help your careeer. They talked about the importance of your internet activity to your prospects of getting hired, and steps you can take to better represent yourself online. Stever Robbins provided a list of guidelines that dispelled some of the “myths” associated with reputation and career in general. Chris Brogan, with a good dose of humor, described how his blogging connected with his career experiences and the importance of the human element in business (e.g. getting acquainted with someone through his Flickr postings before realizing he’s a prominent VC).
2. Using social media to recruit. This discussion was led by Aaron Strout and Todd Defren, who described their own experiences using Facebook, Twitter, blogs, and other channels. The main idea expressed was that several aspects of a person’s online activity are very important in identifying good candidates:
   • what she writes in her blog or microblog (Twitter) and her Facebook and LinkedIn profiles
   • any content about her that comes up in a Google search, including potentially negative material and material written by others about her

The conversation was enlightening. Todd declared that in the PR space, for example, a candidate isn’t likely to be hired or even interviewed without being “well known” on the internet. More generally, the wisdom prevailing as recently as 5 years ago - that one should be careful about making communications visible online - seems to have reversed polarity. Today the greater risk clearly lies in NOT being visible online, with the chances of a potential employer seeing what happened in South Padre last spring break being actually much less if there is significant other material out there in one’s favor.

There were a couple of issue that weren’t well addressed, despite some audience questions along these lines. The discussion remained limited to the PR-oriented purview of many of the speakers (and audience), so the hiring issues discussed were typically those faced by prestigious PR firms. These hires would be expected to be media savvy and these firms would also tend toward elite hiring practices (e.g. “I want to hire the abosolute best 1-2 guys in the field”). The question of hiring other kinds of professionals (e.g. software engineers) was raised, but a solid answer wasn’t really given. The example of hiring Joel Spolsky because of the prominence of his Joel on Software blog suggests either an unrealistic view of the software industry or social media industry myopia. While many software firms would be psyched to hire Joel, every firm must be willing to hire developers who don’t have widely-read tech blogs, and would probably be better served by looking for the “rank & file” contributors to the social media world, the talented engineers who are reading and/or commenting on Joel’s blog. Finally, there was no attention paid to the majority of people not yet actively using social media. Though we might be better off if more of them did, we can’t expect every dental hygienist to have a blog in order to be doing the job well.

These shortcomings in content covered are quite understandable given the goals and audience of the event, and are pointed out as suggestion for further discussion rather than a negative reflection on the event.  Overall the session was excellent and the folks there were very cool.

Thanks to Bryan Person for organizing!

Mark Soper