{"componentChunkName":"component---src-templates-default-template-tsx","path":"/tutorials/tutorial-google-calendar-connector/","result":{"data":{"asciidoc":{"id":"623b5f38-e113-5308-94bf-0b9bcb29e10c","html":"
\n
Table of Contents
\n\n
\n
\n

Overview

\n
\n
\n

In this tutorial, we will show how to write a simple connector for HIRO. We have picked Google Calendar as example application. The connector we develop in this tutorial will read events from a calendar and copy them to the HIRO knowledge core. To keep the connector simple, we will copy the events only in one direction. Synchronization of data will be covered by another tutorial.

\n
\n
\n

The code examples are written in Python, but the HIRO API can be used with any other programming language, as well.

\n
\n
\n
\n\"google\n
\n
Figure 1. Your connector integrates HIRO with Google Calendar
\n
\n
\n

We will perform the following steps to implement the connector:

\n
\n
\n
    \n
  1. \n

    Prepare prerequisites in Google Calendar / Google Cloud

    \n
    2. \n
  2. \n

    Connect with HIRO

    \n
    3. \n
  3. \n

    Fetch events from Google Calendar

    \n
    4. \n
  4. \n

    Write retrieved events to HIRO

    \n
    5. \n
  5. \n

    Synchronization: only write new or updated events

    \n
    6. \n
  6. \n

    Create a loop to continuously poll events

    \n
    7. \n
\n
\n
\n
\n
\n

Prerequisites & Preparations

\n
\n
\n

Preparing a Google service account

\n
\n

We want to use the Google API to read data from a Google calendar. In order to do this, we need user and application credentials, similiar to the HIRO API. The required steps are:\n. Create a google cloud account\n. Create a new project\n. Create Service Account credentials new app/project

\n
\n
\n

If you already have the required accounts and credentials, you may skip to the next section. Otherwise, continue to read the step-by-step\ninstructions.

\n
\n
\n

Create a google cloud account

\n
\n
\n

Go to https://cloud.google.com/ and sign in. If you don’t have a Google Cloud account, you have to register first.

\n
\n
\n

Create a new project

\n
\n
\n

In the Google Cloud Platform, users, credentials, and permissions are organized in projects. We will create a new project \"HIRO-Connector\" in the Google Cloud.

\n
\n
\n

Proceed to the Google Cloud console: https://console.cloud.google.com/ , and click on the project drop-down box on top. Click on \"NEW PROJECT\" to create a new project. Now enter the name of your new project: \"HIRO-Connector\".

\n
\n
\n
\n\"GCP\n
\n
Figure 2. Create new GCP project
\n
\n
\n
\n\"GCP\n
\n
Figure 3. Enter project name
\n
\n
\n

You have created a new Google Cloud project. As next steps, we will create an API key for our application, and a service account.

\n
\n
\n

Create Service Account credentials new app/project

\n
\n
\n

From the Google Cloud Console home screen, select \"Credentials\" from the \"API & Services menu\". Click on \"create credentials\", and then \"Create service account\".

\n
\n
\n
\n\"GCP\n
\n
Figure 4. GCP - APIs & Services
\n
\n
\n
\n\"GCP\n
\n
Figure 5. GCP - New credentials
\n
\n
\n

We enter a new service account name, which we call \"hiro-connector\" and press create. On the next screen select the role as editor and click continue. On the last screen click done.

\n
\n
\n
\n\"GCP\n
\n
Figure 6. GCP - Create service account
\n
\n
\n
\n\"GCP\n
\n
Figure 7. GCP - Edit role
\n
\n
\n

On the service account page, click on the button next to key creation date and press create key.

\n
\n
\n
\n\"GCP\n
\n
Figure 8. GCP - Create private key
\n
\n
\n

The key type should be JSON. Press the \"Create\" button, and you’ll be prompted to save the credentials file to disk. This file contains all necessary credentials data to connect to the API.

\n
\n
\n
\n\"GCP\n
\n
Figure 9. GCP - Download private key
\n
\n
\n

To grant calendar permissions to this service account, you will need the account’s email address later. Look it up in the GCP console while you are still there. It should look like this: hiro-connector@hiro-connector-123456.iam.gserviceaccount.com

\n
\n
\n
\n

Preparing a Google Calendar

\n
\n

We will now create a new Google calendar to work with. You may of course also work with an existing calendar, if you prefer.

\n
\n
\n

Create a new Google calendar\nGo to Google Calendar (https://calendar.google.com/) and create a new calendar (\"Other calendars\" -→ \"+\" -→ \"Create new calendar\". Enter \"HIRO-TEST\" as name.

\n
\n
\n
\n\"new\n
\n
Figure 10. New Google calendar
\n
\n
\n

Next. we enter the calendar setting for the \"HIRO-TEST\" calendar. We share our calendar with the service account hiro-connector@hiro-connector-123456.iam.gserviceaccount.com and provide it edit permissions.

\n
\n
\n
\n\"share\n
\n
Figure 11. Share Google calendar
\n
\n
\n

Copy the calendar ID from the settings page - you will need this later to access the calendar through the API.

\n
\n
\n

Enable Calendar API for the new project (in Google Developer Console)

\n
\n
\n

There is one last step that we have to perform: enable the Calendar API for your new project. We do this in the Google Developer Console: https://console.developers.google.com/

\n
\n
\n

Select the \"HIRO Connector\" project, and click on \"+ ENABLE APIS AND SERVICES\". Select \"Google Calendar API\", and enable it on the next screen. Your are ready now to connect via API to your calendar.

\n
\n
\n
\n\"enable\n
\n
Figure 12. Enable calendar API
\n
\n
\n
\n

Install Python libraries for Google API

\n
\n

You will need two Python libraries for the Google API. In case you do not have them in your environment yet, install them with the pip command.

\n
\n
\n

Google Oauth client:

\n
\n
\n
\n
pip install google-auth
\n
\n
\n
\n

Google Python client library:

\n
\n
\n
\n
pip install google-api-python-client
\n
\n
\n
\n
\n

Preparing HIRO session

\n
\n

Finally, we initiate a HIRO session, as seen in the First Steps with HIRO tutorial:

\n
\n
\n
\n
# Initiate session\nhiro_session = initiate_hiro_session('./hiro_credentials.json')
\n
\n
\n
\n

We will also need the functions create_vertex, update_vertex and create_edge, which we have defined in the Basic Vertex Operations tutorial.

\n
\n
\n
\n
\n
\n

Connecting with the Google API

\n
\n
\n

Before we can read and write data via the Google API, we need to authenticate and create a session with the intended usage scope.\nWe load the service account credentials, which we had downloaded from the Google Cloud Platform Console before (see above), set the scope to 'https://www.googleapis.com/auth/calendar', and create an authenticated session.

\n
\n
\n
\n
# google api imports\nfrom google.oauth2 import service_account\nfrom google.auth.transport.requests import AuthorizedSession\nfrom googleapiclient.discovery import build\n\n# configuration\ngoogle_credentials_file = \"./forward-aura-168008-92a0f1f9cd3a.json\"         #replace with your credentials file!\ngoogle_calendar_id = '7hfea2godbmajrspfp30l23q1o@group.calendar.google.com' #replace with the id of your calendar!\n\n# load credentials and authorize session\ncredentials = service_account.Credentials.from_service_account_file(google_credentials_file)\nscoped_credentials = credentials.with_scopes(['https://www.googleapis.com/auth/calendar'])\nauthed_session = AuthorizedSession(scoped_credentials)
\n
\n
\n
\n

Once we are authenticated and have created a session with the Google API, we can instantiate a specific service for the Calendar API:

\n
\n
\n
\n
# Build Calendar API Service\nservice = build('calendar', 'v3', credentials=scoped_credentials)
\n
\n
\n
\n
\n
\n

Reading calendar data from Google

\n
\n
\n

As next step, we will check that we can read calendar data through the Google API, to verify that we got everything in place correctly. As a test, we print out the calendar name to the console, and retrieve all calendar events.

\n
\n
\n
\n
# get calendar name\ncalendar = service.calendars().get(calendarId=google_calendar_id).execute()\nprint ('Calendar name: ' + calendar['summary']+ '\\n')\n\nprint('Getting all events')\nevents_result = service.events().list(calendarId=google_calendar_id, singleEvents=True, orderBy='startTime').execute()\nevents = events_result.get('items', [])\n\nif not events:\n    print('No upcoming events found.')\nfor event in events:\n    start = event['start'].get('dateTime', event['start'].get('date'))\n    print(start, event['summary'])
\n
\n
\n
\n

We have read data from Google Calendar and printed it out to the console. We are ready for the final step: writing the calendar data to HIRO.

\n
\n
\n
\n
\n

Writing calendar data to HIRO

\n
\n
\n

For writing calendar data to HIRO, we define two helper functions. The first function write_calendar() creates a node in the HIRO Graph, which represents the specific calendar. We add the calendar id as _xid and calendar name as title attribute, so that we know which calendar it is.

\n
\n
\n

The second helper function write_event() creates a vertex of type ogit/Schedule/Event in HIRO for the calendar event from Google. In this example, we write the event id, title, start and end time, description, location, and etag as attributes. You might want to extend this with additional attributes in the future. The new event vertex is connected with a ogit:contains edge to the calendar vertex.

\n
\n
\n
\n
def create_calendar (calendar):\n    vertex_data = {'ogit/title':calendar['summary'],\n                   'ogit/_xid': 'CAL-' + calendar['id'],\n                   '/etag': calendar['etag']\n                  }\n\n    return create_vertex('ogit/Schedule/Calendar', vertex_data)\n\ndef create_event (cal_id, event):\n\n    vertex_data = {'ogit/title':event['summary'],\n                   'ogit/Schedule/startTime': event['start'].get('dateTime', event['start'].get('date', '')),\n                   'ogit/Schedule/endTime': event['end'].get('dateTime', event['end'].get('date','')),\n                   'ogit/_xid': 'CAL-' + cal_id + '-' + event['id'],\n                   '/etag': event['etag']\n                  }\n\n    if 'description' in event:\n        vertex_data['ogit/description'] = event['description']\n    if 'location' in event:\n        vertex_data['ogit/Schedule/location'] = event['location']\n\n    vertex_id = create_vertex('ogit/Schedule/Event', vertex_data)\n    create_edge(cal_id, vertex_id, 'ogit:contains')\n\n    return vertex_id
\n
\n
\n
\n

Now we can retrieve the calendar data from Google again, and write the entries to HIRO. We will first create a calendar item in HIRO. Afterwards, we iterate over all calendar events, copy each event to HIRO, and connect it to the calendar vertex.

\n
\n
\n
\n
# get calendar name\ncalendar = service.calendars().get(calendarId=google_calendar_id).execute()\n\n# create calendar vertex in HIRO\nhiro_cal_id = create_calendar(calendar)\n\n# retrieve all events from Google calendar\nevents_result = service.events().list(calendarId=google_calendar_id, singleEvents=True, orderBy='startTime').execute()\nevents = events_result.get('items', [])\n\nfor event in events:\n    hiro_event_id = create_event (hiro_cal_id, event)
\n
\n
\n
\n

If you check now, you will find your calendar and all events in the HIRO Graph.

\n
\n
\n
\n
\n

Continuous synchronization: repeatedly write new or updated events

\n
\n
\n

Of course, we don’t want to create new data objects every time we check the Google calendar, but we want to update the data in HIRO when there are changes or new events in the Google calendar. Therefore, we need to first check whether there are already nodes in HIRO for the calendar or event; and if they exist, we want to know whether there have been any changes that need to be updated.

\n
\n
\n

HIRO offers a special 'ogit/_xid' attribute, which should be used to specify the external id that the item has in the original data source. If you write for instance the Google event id to this field, you will be able to check whether a specific event is already stored in HIRO or not. There is also a convenience method in the Graph API that facilitates looking up vetices by external id.

\n
\n
\n

External ids are supposed to be unique in the HIRO database. Therefore, we do not just use the calendar or event ids in our example, but add a prefix, so that we reduce the probability of id conflicts. This is a good practice that you might want to adopt already in your first connector.

\n
\n
\n

Google has introduced a nice feature for determining whether a data object has changed: there is an etag attribute, which is assigned a new random value when the object is changed. As we have already copied this attribute to the HIRO vertices before, we can simply check whether the etag has changed. Thanks Google!

\n
\n
\n

We define helper methods to check whether the calendar and events already exist and whether there are changes:

\n
\n
\n
\n
def update_calendar (vertex_id, calendar):\n    vertex_data = {'ogit/title': calendar['summary'],\n                   'ogit/description': calendar['description'],\n                   '/etag': calendar['etag']\n                  }\n\n    return update_vertex(vertex_id, vertex_data)\n\ndef read_vertex_by_xid (xid):\n    response = hiro_session['session'].get(hiro_session['graph_api'] + 'xid/' + xid )\n\n    if response.ok:\n        if len(response.json()['items'])==1:\n           return response.json()['items'][0]\n    else:\n        print (str(response.status_code) + \" - \" + response.text)\n        print ('Failed to read vertex.')\n    return None\n\ndef create_or_update_calendar (calendar):\n    xid = 'CAL-' + calendar['id']\n    etag = calendar['etag']\n\n    cal_vertex =  read_vertex_by_xid (xid)\n\n    if cal_vertex == None:\n        return create_calendar (calendar)\n    else:\n        print (cal_vertex)\n        hiro_etag = cal_vertex.get('/etag', '')\n        vertex_id = cal_vertex ['ogit/_id']\n        if etag != hiro_etag:\n            update_calendar (vertex_id, calendar)\n        return vertex_id\n\n\ndef update_event (vertex_id, cal_id, event):\n    vertex_data = {'ogit/title':event['summary'],\n                   'ogit/Schedule/startTime': event['start'].get('dateTime', event['start'].get('date', '')),\n                   'ogit/Schedule/endTime': event['end'].get('dateTime', event['end'].get('date','')),\n                   '/etag': event['etag']\n                  }\n\n    if 'description' in event:\n        vertex_data['ogit/description'] = event['description']\n    if 'location' in event:\n        vertex_data['ogit/Schedule/location'] = event['location']\n\n    return update_vertex(vertex_id, vertex_data)\n\n\ndef create_or_update_event (cal_id, event):\n    xid = 'CAL-' + cal_id + '-' + event['id']\n    etag = event['etag']\n\n    event_vertex =  read_vertex_by_xid (xid)\n\n    if event_vertex == None:\n        create_event (cal_id, event)\n    else:\n        hiro_etag = event_vertex.get('/etag', '')\n        vertex_id = event_vertex ['ogit/_id']\n        if etag != hiro_etag:\n            update_event (vertex_id, cal_id, event)
\n
\n
\n
\n

And now we can adapt our synchronization routine to utilize the new methods:

\n
\n
\n
\n
import time\n\n# get calendar name\ncalendar = service.calendars().get(calendarId=google_calendar_id).execute()\n\n# create calendar vertex in HIRO\nhiro_cal_id = create_or_update_calendar(calendar)\n\n#endless loop\nwhile True:\n    # retrieve all events from Google calendar\n    events_result = service.events().list(calendarId=google_calendar_id, singleEvents=True, orderBy='startTime').execute()\n    events = events_result.get('items', [])\n    for event in events:\n        hiro_event_id = create_or_update_event (hiro_cal_id, event)\n\n    time.sleep(60)
\n
\n
\n
\n

Try it out by adding new events and changing events in your Google calendar. Within the endless loop, the connector code checks once per minute for new or changed data. Only new or changed events will be written to the HIRO Graph.

\n
\n
\n
Calendar event in HIRO Graph
\n
\n
{\n  \"/etag\" : \"3130664879208000\",\n  \"ogit/Schedule/endTime\" : \"2019-08-09T10:30:00+02:00\",\n  \"ogit/Schedule/location\" : \"somewhere else\",\n  \"ogit/Schedule/startTime\" : \"2019-08-09T10:00:00+02:00\",\n  \"ogit/_created-on\" : \"Fri, 09 Aug 2019 06:31:28 GMT\",\n  \"ogit/_creator\" : \"user1@academy.local\",\n  \"ogit/_creator-app\" : \"cjix82rxi000gu473w5kvkpqv\",\n  \"ogit/_graphtype\" : \"vertex\",\n  \"ogit/_id\" : \"cjz3qffjoci6bgp027kuzwx5g\",\n  \"ogit/_is-deleted\" : \"false\",\n  \"ogit/_modified-by\" : \"user1@academy.local\",\n  \"ogit/_modified-by-app\" : \"cjix82rxi000gu473w5kvkpqv\",\n  \"ogit/_modified-on\" : \"Fri, 09 Aug 2019 06:34:30 GMT\",\n  \"ogit/_owner\" : \"academy.local\",\n  \"ogit/_type\" : \"ogit/Schedule/Event\",\n  \"ogit/_v\" : \"3\",\n  \"ogit/_v-id\" : \"1565332470438-Swxvdw\",\n  \"ogit/_xid\" : \"CAL-cjz1frals2bn2gp02iq46j9ex,-57f7qq54o0hf25sr3jcereti4b\",\n  \"ogit/title\" : \"TEST\"\n}
\n
\n
\n
\n
\n
\n

Summary

\n
\n
\n

In this tutorial, we have implemented our first HIRO connector.The connector imports calendar events from a Google calendar into the HIRO Graph.

\n
\n
\n

We have performed the following steps for implementing this connector:

\n
\n
\n
    \n
  • \n

    Created a new project and a service account in Google Cloud Platform

    \n
    • \n
  • \n

    Created a new Google Calendar and configured permissions

    \n
    • \n
  • \n

    Established an authenticated session with HIRO

    \n
    • \n
  • \n

    Fetched events from Google Calendar

    \n
    • \n
  • \n

    Written retrieved calendar events to HIRO

    \n
    • \n
  • \n

    Create a loop to continuously poll events and write changes to HIRO

    \n
    • \n
\n
\n
\n

As you probably have noticed, we have copied only a subset of the calendar event attributes to HIRO. You may extend the code to make more attributes available.

\n
\n
\n

Now that you have calendar data in HIRO, you can automate some tasks to manage your calendar. If you haven’t read the tutorial about creating automation tasks yet, we suggest to continue there.

\n
\n
\n
","document":{"main":"Tutorial","title":"Tutorial: Developing a Connector for HIRO","subtitle":"Developing a Connector for HIRO"},"fields":{"toc":true,"location":["tutorials","tutorial-google-calendar-connector"]}},"sidebarYaml":{"id":"0acf6fc1-95eb-5854-8f81-d5561b07a582","showIndex":null}},"pageContext":{"id":"623b5f38-e113-5308-94bf-0b9bcb29e10c","parent":"tutorials"}},"staticQueryHashes":["1010459453","1010459453","2356112386","2356112386","2603905930","2603905930","3026652197","3026652197","3167850324","3167850324","63159454","63159454"]}