Introduction
“Who’s That?” is a pervasive application that aims to connect North Quad residents by displaying relevant profiles in public spaces and encouraging interactions. Below is the general workflow of the application:
- A resident first create a profile based on key characteristics (i.e. year, major, floor, and interests) identified in our research with North Quad residents. The resident should also choose one of three kinds of interactions, namely “Contact me”, “Learn more”, and “Play a game” that can be initiated by other residents through QR codes within the profile.
- On the monitors in the elevator lobby of each floor of the residential hall, the resident’s profile will be displayed randomly, with a dynamic tag cloud of these characteristics.
- The tag cloud dynamically responds to other residents as they pass in front of the monitor by enlarging relevant (or matching) tags and minimizing irrelevant tags.
- Interested residents then use the smartphone app to capture the QR code to initiate the interaction. “Contact me” will send a message to the profile owner. “Learn more” will redirect user to a website specified by the owner. “Play a game” will initiate a web-based game, and in the end of the game they will be given an option to send a message to the owner.
- The profile creator login to the web application to see the statistics of the interactions and respond to messages from other residents.
To users, our system has three main components:
- Pervasive displays that detect user presence, show relevant and engaging content and initiates interactions with the profile
- Smartphone applications that continue the interactions on personal devices
- A web application that enable profile creation, update and monitoring as well as responding to incoming callouts.
To the future system builders, the system is composed of software and physical objects. The software part can be further categorized as:
- Local servers that process data and control the displays
- Presence detectors that detect the presence of users and send the data to core application to process
- Core application that receives data from presence detector, recommends profiles and dynamically changes the tags
- A web server that facilitates creation of profiles, stores data, handles data requests from local servers and responds to QR-initiated interactions
- Web application that assists profile creation, shows statistics and handles messaging
- Mobile web application that responds to QR-initiated interactions
- Database that stores user profiles, statistics and communication history.
- Smartphone applications that capture QR code and login users to the system.
The following paragraphs present what we have achieved, what components are missing in our prototype and our suggestions for future implementation. The functions are marked as “Gold”, “Silver”, and “Bronze” version (▊Gold / ▊Silver / ▊Bronze). From “Bronze” to “Gold”, we progressively step forward from the basic implementation and offer ideal and more compelling versions of the grand vision. In other words, functions that are marked “Bronze” are the essential bricks to build the preliminary version of the deployable system, whereas “Gold” functions are more ideal but also need more resources to implement.
1. Software Development
1.1 Local Server
Local servers process data and control pervasive displays.
1.1.1 Core Application
The core application is currently built in Flash/Actionscript. It is intended to perform a series of key functions, some of which some have been partially built out. Those functions, their current status, and the work required to complete them are listed below. Please note that some of the functions anticipated for a final system would have to be built out as complementary services in other languages such as Java or C#.
▊1.1.1.1 Construct and Display User Profiles
Current Status: The current system constructs user profiles from two elements: a static background image and a dynamic tag cloud that sits on top of it. The background image is pulled from a file stored locally and accessed by file name, and the tag cloud is populated with a list of tag words. Both pieces of information are stored in a static list of “user data objects” that is hard-coded into the application (“UserData.as” for the structure, “UserProfiles.fla” for the content).
Final Status: A complete version will remove all hard-coding of profile content and will pull all user data from a database rather than a pre-written, static list.
Profiles will be comprised of:
- A background image, accessed by file name
- List of keywords for the tag cloud
- “Name” text field
- “Academic year” text field
- “Area of study” text field
- “Call to action” text field
- QR code image, accessed by file name
The database that stores this information for every user must be designed and built, along with the procedures to pull data as needed. In addition, the ActionScript class that organizes and displays this information must be rewritten to accommodate the five additional fields (see “UserProfile.as”). Then, data pulled from the database can be used to initialize new user profiles with no manual intervention required.
▊+ ▊1.1.1.2 Manage Profile Rotation
Current Status: A timer is set up to rotate profiles at any rate desired, as indicated by a hard-coded number measured in milliseconds. In addition, the system resets this timer upon detection of a user presence in order to minimize the chance of profile rotation while the user is engaged with the content (see “UserProfiles.fla”). User profiles are switched by populating two profiles at once and in alternating fashion, hiding one while showing the other.
Final Status: In addition to the features listed above, a visible timer must be added to warn the user of a pending profile change. Further, the process for rotating profiles should be updated to add a brief interim state in which the screen fades to black between profiles.
▊Bonus: If touch screens are installed in the appropriate locations, functionality to pause, rewind and fast-forward must be mapped to the button graphics on the interface. Fast-forward and rewind functionality is currently tied to keyboard presses for the arrow keys (see “UserProfiles.fla”).
▊1.1.1.3 Animate Tag Cloud Based on Common Keyword Tags
Current Status: The system compares the list of tags in the currently displayed user profile to the tags associated with the detected user (see “UserProfile.as”). The system then magnifies the font size for all tags that are an exact match and shrinks the size for all unmatched words. If no matches are detected, then no animation occurs. When tag animation does occur, the system rechecks for collisions between tags and attempts to adjust tag positioning (see “TagCloud.as”).
Final Status: Assuming that user presence detection is available, the solution described above will work properly as a rudimentary fix. However, a more effective version would incorporate two changes. First, the functions used to animate and adjust tag location must be changed to ensure zero tag overlap. More robust freeware solutions to this problem are available, like an ActionScript implementation of Wordle (found here http://int3ractive.blogspot.com/2011/02/as3-wordle-tag-cloud.html). Second, a natural language processing solution is required to ensure that very similar tags are recognized as being matches despite small differences in phrasing or spelling. This could be a significant technical challenge and will require additional research. Some ideas to initiate that effort follow below.
Key Technical Challenge – Natural Language Processing: Wordnet is a large lexical database of English developed at Princeton University. It could form the basis of a solution that recognizes similar semantic meaning between tags. The system recognizes sets of “cognative synonyms”, and inter-links those words based on both cognitive and lexical relationships. Details can be found at http://wordnet.princeton.edu/.
A number of open source natural language processing libraries take advantage of the Wordnet database. Those include the following, labeled by programming language:
- Java: http://sourceforge.net/projects/opennlp/
- C#: http://sharpnlp.codeplex.com/
- Python: http://www.nltk.org/
▊1.1.1.4 Determine Which Profiles are Displayed To the User
Current Status: The system pulls profile data from a hard-coded list that is presented in no specific order. (see “UserProfiles.fla”) A similar non-ordered but usable solution could be implemented with a database.
Final Status: Ultimately, the application will only succeed in helping to connect residents of North Quad if it is able to present individuals interests, work or other characteristics are perceived as relevant to the users who view them. Hence, it is critical that a final version of the system feature a recommender system that selects profiles to display based on the relevance of tag cloud words or the call to action. Like the natural language processing problem, a robust recommender system is a significant technical challenge. Rules that would dictate the behavior of such a system are described below.
Key Technical Challenge – Recommender System: The recommender system should be comprised of four component that each take into account aspects of past user behavior.
- SVD Algorithm – The SVD algorithm produces scores for each “user/profile” pair. The algorithm would consider which profiles the user has interacted with in the past, and would calculate a set of “features” that a common to those profiles. It then does the same across other users and finds new profiles that were interacted with by users who prefer a similar “feature” set. The scores generated (“profile match scores”) indicate how good a fit each new profile is for the user based on the “features” of these past interactions.
- Keyword Tag Matching – The scores generated by SVD should then be heavily re-weighted based on the number of matching keyword tags between users. While the amount of re-weighting per tag match should be determined based on further research, this explanation will assume that every individual tag match could as much as double a profile match score. The exact amount of impact for every tag match should be determined by two factors.
- Tag frequency – Based on our user research, tag matches for extremely uncommon interests are much more compelling for users than matches on common interests. Hence, the impact of any tag match on the profile match score should be weighted by the frequency of that tag across all user profiles.
- Tag type – Profile keyword tags will be divided into categories like “interests” and “skills” when they are entered by the user. It may be that the category in which the user enters the most tags gives a hint of the social goals that user is pursuing through the system. The recommender system should give extra weight to tag matches in the category preferred by the user in their own profile. This weighting should be heavily tested to determine if it has the desired impact.
- User Blocking – Users will have the option to block other individual users from seeing their profile. Once final matching scores have been calculated, the recommender system should remove any profile from the recommendation list in which one party is in the other’s blocked list.
- Randomization Factor – Final recommendations should be generated by using a randomization factor to select from the recommendation list, with probability of selection determined by each profile’s final match score.
1.1.2 Presence Detector
The Presence Detector has been implemented using VC#. The function of the Presence Detector is three-fold:
- Displays are reactive to the presence/absence of people
- The presence of users is used to determine the profiles to be pulled up and displayed
- Tags are animated based on those that match with the detected user’s tags
Current Status: The C# application listens for Bluetooth signals. On detecting one or more signals, the application transmits the MAC addresses of the detected Bluetooth devices to the Flash application.
Technical challenges with the approach:
- Users should have Bluetooth enabled on their phones
- Bluetooth device discoverability is active for only 120 seconds and has to be reset periodically
- Permanently having Bluetooth enabled would drain a lot of power from the phones
Final Status: We have identified possible alternatives to using Bluetooth for presence detection
- ▊When touchscreen monitors are used for displaying profiles to users, in lieu of bluetooth, user presence can be detected by users explicitly logging into the system through a touchscreen interface. A shortcoming with this approach is that there may a breakdown when there are multiple users in the vicinity. Users may not want to explicitly interact with the monitors in terms of logging in, due to privacy issues.
- ▊The system could employ facial detection through the likes of devices like Kinect, and identify the user in order to display relevant profiles and animate tags. But, facial detection requires the user to look at the camera directly. This could beat the purpose of animating tags for catching viewer’s attention. Also, users may have privacy concerns about cameras capturing and storing their images.
- ▊Smartphones are now equipped with RFID tags and can be used for Near Field Communication (NFC.) In the near future, active RFID tags would be present in all smart phones. The Presence Detector would not have to constantly poll for devices. On the contrary, the active RFID tag could initiate communication with the Presence Detector.
1.2 Web Server
The web server facilitates creation of profiles, stores data, handles data requests from local servers and responds to QR-initiated interactions.
1.2.1 Web Application
‘Who’s that?’ web application is used to create and edit user profiles and to display interaction statistics to the user.
Current Status: Users enter their name, basic details and tags that describe their personality and interests into the system through a web-based form. They also upload high-resolution images for profile pictures. The information, tags and images are laid out as profiles and calls-to-action converted into QR-codes, manually by the application creators.
Final Status:
- ▊ Users will create, update, and monitor their profile through a web application. Creating a profile will involve choosing a high-resolution image, entering one’s name and basic details, and filling out a list of tags which describe one’s personality. Users will also have to choose a ‘call-to-action’ for others to access through QR-codes. A call to action can be a web-link, a prompt to send a message, or a simple game. Users will customize themes and layout of different elements on their profiles. A dashboard interface will display information about the number of QR-code interactions with the user’s profiles, profile views and received messages, if any, from profile viewers.
- ▊ Instead of making users enter information that is required during profile creation for other applications, the system will provide the option to reuse profile information from existing social network accounts. A light-weight dashboard will be implemented on the mobile interface for statistics about messaging and gaming. The web application will support a leaderboard for profile viewers who have played the game created by the user.
1.2.2 Mobile Web Application
Current Status: The web interface is ready to use and can respond to user input. However, because we do not have a database yet, all the user-related information is hard-coded into the HTML.
Final Status: The mobile web application should connect to the database and fetch information from it. When users send back messages, the application should also update the database.
▊1.2.2.2 Respond to “Play a Game” Interaction
Current Status: We have a game “Two truth and a Lie” built in the mobile web application. The interface and interactions on the interface are ready to use and can respond to user input. But because we do not have a database yet, all the user-related information and the game itself is hard-coded into the HTML.
Final Status: The mobile web application should connect to the database and fetch user information from it. The application should also communicate with database to determine the wrong/right answers. When users send back messages, the application should also update the database.
▊1.2.2.3 Receive user MAC Address to Automatically Login Users
Current Status: This function is not built yet.
Final Status: The mobile web application should receive MAC information from the URL (or HTTP post method) so that users do not need to explicitly login to the system
1.2.3 Database
Current Status: The system uses a light-weight datastructure in the place of a database, for storing mapping information about device MAC addresses and user profiles.
Final Status: The system will employ a full fledged database server for storing information about user profiles, user interactions and communication history. The database will be used for mapping detected presence to user profiles. Information about profile views and user interactions for different calls to action will be stored in the database for providing additional features, like the leaderboard for games, as discussed above. These information will also be used by the recommender system to display relevant profiles to users bases on previous views and completed interactions.
1.3 Smartphone Application
▊1.3.1 Capture QR Codes
Current Status: During our prototyping phase, we modified a simple QR-scanning application so that it would append test to any web-link found in the QR code. The intent was that the system could use any incoming URL requests to determine which user was looking at (and scanning) the profile. Technical problems with Bluetooth led us to manually trigger the presence detection during testing.
Final Status: It is plausible that the system could work without a smart-phone application, however, under ideal conditions, it might include the following features.
- The application would need to present a one-touch step to QR-scanning.
- The user should be able to start the application and begin scanning immediately.
- The application should also have a customized header or some other indicator that it is the “Who’s that?” application.
▊1.3.2 Automatically Log-in Users
Current Status: This function is not currently implemented.
Final Status: Ideally, the application would append the phone’s WIFI or Bluetooth MAC address, which the web application could use to look up a username in the database and identify the user so that users don’t need to enter username and password.
▊1.3.3 Receiving Notifications
Current Status: This function is not currently implemented.
Final Status: The application should also have a means of receiving notifications from a central server. Essentially whenever a user receives a message from someone in the Who’s That?” system, that should get a pop-up (or status bar) notification via the application. When they select the notification, it should send the user to the mobile web-interface.
▊1.3.4 Cross-platform Support
Current Status: The current application only supports Android.
Final Status: The application should have iOS, Android, Symbian, Blackberry version.
▊1.3.5 Automatically control Bluetooth discoverability
Current Status: This function is not currently implemented.
Final Status: If Bluetooth is used as a presence detection method, the application should constantly check for MAC address of the display servers in our system. Once it detects one, the application should turn on the phone’s discoverability, hence allowing the system to detect and react to the user. This is necessary because phones need to be discoverable for the system to detect them, and discoverability only lasts for 120 seconds at a time on most devices.
2 Physical Requirements
2.1 Hardware
2.1.1 Displays
▊2.1.1.1 High-resolution Monitors
Displays will need to have large screens, 27-36 inches, capable of displaying high resolutions (720p-1080p) and Displays should be positioned at average shoulder level to allow for comfortable viewing and to avoid making the user strain themselves when scanning QR-codes or using touch-screen.
Current status: Implemented.
2.1.1.2 Touch Screen Support
Ideally displays should be equipped with touch-screen capabilities to allow use of the “Pause” and “Back” buttons displayed on screen.
Current Status: Not yet implemented
Final Status: Displays should have either:
- ▊A touch-screen overlay, for touch-to-mouse control
- ▊Full touch-screen with multi-touch
2.1.2 Local Server
The local device would be a small, compact, low-power device. The device would need moderate storage and processing requirements. It would also need Bluetooth capabilities and internet connection. The Apple Mac-Mini could be a viable option.
Current Status: Not yet implemented; prototypes were hosted on personal laptops.
Final Status: ▊The Apple Mac-Mini would be a viable option.
2.2 Location:
Displays should be setup in area of high-traffic and high visibility, but with care to avoid fire hazards and irritating users. System hardware should be discretely installed so as to avoid showing unsightly cables and hardware. The system should appear to be little more than a display.
Suggested areas for installation include elevator lobbies, hallways, and foyers. The screens should be in such a place that the user is not inhibited to interact with the display by physical or social obstacles. Thus, installations should be away from tables, corners, or cluttered areas, and should not place the user in a conspicuous position when they interact with it (i.e. in front of a group of people).
- ▊In elevator lobbies, on existing monitors
- ▊In North Quad entrance foyers, on existing monitor
- ▊New installations interspersed throughout long corridors
Who's That? 