Project SMILE Winner of SAP TechEd 2015

We won the Las Vegas Demojam with an application we built in just 32 hours. We presented it on stage to give you an idea of what it can do but we didn’t give you a complete picture of exactly what went into it.

The Brief

Our brief was to reduce employees engaging in bad habits that will harm their health at work. We built an application that provided an offering to the end employee as a native mobile app, a web interface for the company health therapist, a sensor for the desk to capture the data and a backend to tie all this together with analysis capabilities. The purpose was to monitor the following employee attributes:

  • Temperature of Employee
  • Movement of Employee at desk
  • Hydration of Employee

The Hardware

Keytree_SMILE_Demojam_thermopileThe key capture of the data was the hardware selection. We settled on a thermopile. Thermopile sensors monitor temperatures of what they are looking at by analysis the infra-red being emitted. It performed 64 unique temperature measurements 15 times per second which was read by an Arduino UNO before the raw data was pushed to SAP HANA. To give you an idea of data scales this gives us approximately 960 readings per second and just shy of 83 millions unique temperatures per day. The actual electronic circuit itself required the addition of passives, regulators, pull ups and a few other components so when we built it on a breadboard it did look like a tangle of wires but it did provide clean data.

For the purposes of the demo we mounted the sensor on top of a LG HD webcam and then coordinated the field of view of the webcam with the field of view of the sensor so that we were able to demonstrate what the hardware was actually doing. For actual use the webcam would not be required.

We also put in Bluetooth ibeacon technology to allow user validation at a particular desk. This means that the user has the SMILE app installed when they come near their device can see the ibeacon and then automatically sign them in when they are close to the sensor.

The Logic

With all of this data being captured from the hardware the processing and analysis is the next stage of the application. On each frame we look for significant areas. A significant area is defined as a region of temperature readings that differs from the ambient temperature and is of a particular size. To do this we actually pulled an image processing technique for blob detection and analysis, these blobs were then compared to known entities or signatures of entities so we had a signature of a human face that was between 35-40 degrees and of a particular size and we had similar for cold and hot drinks. Each of these detected signatures have a calculated weighted center point, magnitude and maximum value. At this stage we have detection of a human being present and their temperature. With some research we discovered that the forehead is a reasonably accurate body temperature, but is on average two degrees below the core body temperature as it is the skin surface temperature we are monitoring but the forehead has blood flow close to the surface.

With the rate at which the sensor provides data we look at the difference between frames. Once these blobs start to be analyzed in movement we were able to calculate their interactions. So now we have the movement of an individual’s head at a desk and then when a drink is put to their mouth.

For the organisation level we are not interested in the individual values of these we just want one metric to measure the health of an employee so these metrics are combined into a single SMILE score ranging from 0 to 100 of how healthy the individual is and then it can be averaged over the course of a day. So SAP HANA has taken 83 million data points and worked out one value of how healthy a person is.

As an additional level we took the temperature of an individual over time and compared it with known temperature variance of certain illnesses such as influenza so that a person can be warned if they should go and preemptively get themselves checked out by the doctor.

The Backend

We needed to tie all of the application together in a fast reliable way. We set out with NodeJS using the SAP HDB connector, websockets (via Socket IO) and rest service (via HTTP). This gave us a platform that can connect data streams over the full duplex connection methods or run one of requests all connected to the data stores in SAP HANA.

One of the aspects we did foresee early on is that writing the raw SQL statements to pull data from our HANA model which was created in the SAP HANA Tools for Eclipse wasn’t going to be as fast as it needed to be. Therefore, we wrote a library that creates the statements much like Squel does for MySQL but for HANA. This library was more time consuming to get working for the first few queries we needed in this application but then it meant that we could perform retrievals and queries in HANA using a hand full of lines of code in NodeJS.

The rest service used express library with our own middleware tier to process requests.

The web sockets used socket io so that machines can join particular data feeds and streams of data using a pub/sub and room methodology.

The User Experience

Keytree_SMILE_native_iphone_applicationThis experience was a custom designed and custom UX native mobile app which was geared around give user’s goals. For instance, that you need to drink water 5 times a day or you need to move more than a particular threshold. The iPhone app connects to the data feeds from the backend analysis directly on the web sockets when the app is running and via apple push notifications (APNS) when it isn’t. The mobile device receives notifications when you check in to the sensor (using the iBeacon), your temperature, movement and hydration which are shown in real-time within the app and actions performed such as when you take a drink so no need to touch the screen for it to be acknowledged.

The Organisation Experience

Keytree_SMILE_DashboardFor this we went with a Web UI built in ReactJS which consumed the websocket and rest services for data acquisition. This data was then delivered to the UI within graphs, floor plans and alert lists so it was easy to see what was going on.

This means that easily a organization health therapist can see which desks people are at, their temperature and any notifications that are of concern for instance a user has drunk too much coffee or someone is too stationary.

The End Result

The complete solution provides a comprehensive health system for an organization with benefits and motivations for both the individual users and the organization.

In the 32 hours we build a mobile native app, web interface, electronic hardware and SAP HANA data model (with a custom built library on top). We are looking at taking this hardware forward and making it into a product and give everyone a chance to SMILE.

Bookmark and Share

Presentation at SAP Dcode 2014 AR and VR for Enterprise. How to change the World?

SAP DCode 2014 Tec119 How to Change the World Augmented and Virtual Reality in Enterprise I am giving a presentation at SAP Dcode (originally teched) 2014 on the following topic:

AR and VR How to change the world?

How can augmented reality and virtual reality make a difference? What technologies are available today and coming? Find out from three-time DemoJam winners who pioneered smart glasses AR connected to SAP HANA. This extends going beyond 2D interfaces using 3D, augmented reality, virtual reality, and more connected to SAP providing next-generation applications.

I look forward to seeing you there. Check out the session here.

Bookmark and Share

Oculus Rift for Enterprise – HANADeck

HANADeck is a prototype application we developed that harnesses the Oculus Rift, Leap Motion, Microsoft Kinect and Unity3d for an immersive 3D virtual world which connected with SAP HANA created an enterprise application for data and environment visualisation.

So why use gaming technology for enterprise visualisation? We work in 3D, it is much easier to understand information when it is shown in a way that resembles the real world. Especially with the drive for gesture control and really immersive experiences it is more important than ever to look for truly immersive experiences.

HANADeck uses the display of the 7 inch Oculus Rift with Stereoscopic vision, which delivers the 3D environment. The user is tracked by a microsoft kinect, as this provides the lateral and vertical movement to the virtual environment. It was also found to reduce the motion sickness some people experienced with the Rift as the system is providing motion feedback to the user. Also a Leap Motion controller was mounted on the front of the Rift at 45 degress, which works well at tracking the user’s hands and providing an interaction point. Between the Leap Motion, Microsoft Kinect and Oculus Rift there are all the parts to track and deliver the immersive experience to the user.

The experience focused on in this enterprise demo is taking the visualisations of power lines within the UK network. This is sitting on real world data and purpose is to allow the engineer to travel to electricity towers a lot more quickly and safely.

Bookmark and Share

Uninet – Demojam 2013 Keytree Labs

Our second demojam entry involve making every surface around you an interactive human interface to you business activities.

Smart Fridge / Smart Refrigerator – a transparent multi touch surface on the front of the fridge with controlled internal RGB lighting and lock. You can fill out everything from your expense report to your leave requests on it, whilst getting you one of the sunshine drinks.

Smart Window – Controlled with leap motion, Microsoft kinect you can do similar on any window as you look out.

Bookmark and Share

HANADeck – Demojam 2013 concept

At Keytree labs we have been working hard at for SAP Demojam 2013. We are putting forward two ideas and our first one.

HANADeck is virtual reality experience powered by the Oculus Rift, Leap Motion and Microsoft Kinect. This means that a user can walk around and manipulate with their hands a virtual reality space. For this use case we are using the National Grid pylon database. A service engineer can inspect the data associated with any tower on the network remotely.

Bookmark and Share

Pebble Watch SAP and Keytree Designs

So I have had my pebble watch for a few months and have finally got round to creating some faces. Surprisingly simple and straight forward. At Keytree we work closely with SAP products so I thought I would create one for both Keytree and a fan based one for when working with SAP products.

Keytree Pebble Watch Design SAP Pebble Fan Design Watch

Now for building an app….

I make no claim over logos used. These have been designed as fan material.

Bookmark and Share

Augmented World Expo 2013 – (AWE 2013)

I attended the Augmented World Expo 2013 (AWE).  This event provided a meeting point for augmented reality hardware manufacturers to software SDK developers and from app developers to robotics uses of AR and was held in the Santa Clara Convention Centre in California.

At the event Nic Doodson and I (Will Powell) presented Keytree CEO Vision within the business side of presentations. We presented just after Aiaz Kazi from SAP. Keytree’s CEO Vision, for which I was lead developer on, demonstrates the real possibilities for ‘Minority Report’ like interfaces as they become commonplace and has since been multi award winning.

The event also provided me with a chance to look at the emerging technologies to the market in this rapidly expanding field. Here are just some of the technologies I tried:

Augmented World Expo 2013 Glasses - Will Powell

Google Glass – This is definitely one of the products with the most buzz around it. It has a brilliant build quality and it is understandable that this is only in developer’s hands because the functionality needs to be extended in order to be more consumer facing. However combining this with our expertise, we can deliver enterprise applications on this kind of device.

GlassUp – A neat product with a great price point. Only a single monocular display where a small amount of the glass shows a projected image in monochrome, but very lightweight and unobtrusive. This will be on Kickstarter shortly.

Laster Technologies – There are several products created by Laster Technologies, which were very impressive with the quality of the display screens. I was impressed especially by the contrast with the lit and non-lit areas of the screen. It made the marker-less experience very compelling for the ski goggle’s solution. I look forward to seeing the binocular version of the monocular glasses as well.

Vuzix Star and M100 – are both products we have been tracking with our concept video earlier in the year for the M100 and our use of Star 1200 XL and XLD are great additions to the line up.

Epson Moverio BT-100 – aside from being quite bulky give a good display and with the compact Android power unit are very portable and require no additional processing as the other glasses from Laster and Vuzix do.

Alongside these hardware manufacturers there are the software creators such as Vuforia and Metaio, which were showing off some very impressive demos.

Overall AWE2013 was a fantastic event to both view and demonstrate AR concepts, software and hardware, which left excitement and anticipation of what’s coming next.

Bookmark and Share

CEO Vision wins Gaming and Lifestyle award at Innovation World Cup Wearable Technology Conference 2013

wearable technology award_ceo vision will powell keytree

In a competition with over 300 innovation entries at the Innovation World Cup held during the Wearable Technology Conference in Munich 2013 CEO Vision won the Gaming and Lifestyle category and narrowly missed out finishing runner up to the overall winner Recon Instruments. This is a great achievement.

 

Bookmark and Share

CEO Vision Generation 2 – Demo Jam Winner – Hand Tracking, Augmented Reality, Kinects x 2 and 1.2 billion database rows

CEO Vision generation 2 is a completely new build of CEO Vision with a vast array of improvements. The system now provides a fully interactive, precise working environment.

Continue reading “CEO Vision Generation 2 – Demo Jam Winner – Hand Tracking, Augmented Reality, Kinects x 2 and 1.2 billion database rows” »

Bookmark and Share