Data uitwisseling via webservices

Bouw je eigen API

by Peter Martin / @pe7er

Joomladagen Nederland 2017 in Zeist,
zondag 2 april 2017

Webservice

Google Maps - Website

Via User Interface
(browser)
https://www.google.nl/ maps/place/ Woudenbergseweg+54, +3707HX+Zeist

Google Maps - Webservice

Via Application Protocol Interface
https://maps.googleapis.com/ maps/api/geocode/json?sensor=false&address= Woudenbergseweg%2054 %203707HX%20Zeist

{
"results" : [
 {
  "address_components" : [
  {
    "long_name" : "54",
    "short_name" : "54",
    "types" : [ "street_number" ]
  }
  ],
  "geometry" : {
    "location" : {
    "lat" : 52.086062,
    "lng" : 5.281954199999999
  },

Geschiedenis

1999 - SOAP (Simple Object Access Protocol)
+ WSDL (Web Services Description Language) (voorbeeld)
(Microsoft: Dave Winer, Don Box, Bob Atkinson, Mohsen Al-Ghosein)

SOAP - WSDL 1/4

SOAP - WSDL 2/4

SOAP - WSDL 3/4

SOAP - WSDL 4/4

	WSDL File for HelloService
	
		
	

						

Geschiedenis

1999 - SOAP (Simple Object Access Protocol)
+ WSDL (Web Services Description Language) (voorbeeld)
(Microsoft: Dave Winer, Don Box, Bob Atkinson, Mohsen Al-Ghosein)

1998 - XML-RPC (XML via Remote Procedure Call)
(Dave Winer)

Geschiedenis

1999 - SOAP (Simple Object Access Protocol)
+ WSDL (Web Services Description Language) (voorbeeld)
(Microsoft: Dave Winer, Don Box, Bob Atkinson, Mohsen Al-Ghosein)

1998 - XML-RPC (XML via Remote Procedure Call)
(Dave Winer)

2000 - REST (REpresentational State Transfer)
ook wel RESTful Web services
(Roy Fielding: Proefschrift)

REST

Eenvoudige uitleg

INPUT: specifieke URL +
HTTP requests (POST, GET, PUT, DELETE, etc)

OUTPUT: HTTP status code +
gestructureerde data (bijv JSON)

6 basis principes

1. Client-Server
2. Stateless
3. Cacheable
4. Layered system
5. Code on demand (optional)
6. Uniform interface

REST: 1. Client-Server

Separation of concerns

Scheiding UI & Data:
* portabiliteit: verschillende UIs
* schaalbaarheid: achterliggende data / servers
* UI & API los van elkaar kunnen ontwikkelen

REST: 2. Stateless

Server bewaart geen context (geen cookies)
Elk request van Client bevat alle data
Elk response van Server bevat alle data

REST: 3. Cacheable

Response van GET is cacheable (POST etc niet)
Clients en servers regelen onderling cache

REST: 4. Layered system

Client rechtstreeks Server
of via tussenstappen (Proxy, load balancer, etc)

REST: 5. Code on demand (optional)

Servers kunnen functionaliteit veranderen door code te verstrekken aan Client (bijv JavaScript).

REST: 6. Uniform interface

HATEOAS - Hypermedia as the Engine of Application State

Response bevat alleen hyperlinks voor mogelijke acties

Mijn webservice

2014 - 2016: co-werkplek
Hubbels.net, Nijmegen

Een woensdag in 2015 - iets langer haar

Remko, mijn co-werkplek collega

...had Lunch

No lunch for me!

Wie is er aanwezig?

Website met webapplicatie?
    aankomst? -> log in
    vertrek? -> log out

-> extra handelingen
dus onbetrouwbaar

Wat doe jij als eerste
op je werk?

behalve...
       licht aan
       jas uit
       koffie
       muziek aan
       potje tafelvoetbal

verbinding maken op
laptop of
mobile
met Wi-Fi (LAN)

Raspberry Pi

Raspberry Pi

verbonden met LAN

bash script verzamelt
    MAC adressen
    van alle apparaten

    stuurt die naar
    externe webservice

Waarom een Webservice?

    Waarom niet?

    Omdat het kan?

Webservices

28 oktober 2015
REST API in WordPress 4.4

Augustus 2015
mijn experimentje

op Joomla 3.4.3
(nog) geen REST API

Webservice

SLIM Framework
extern PHP script; RESTFull
    verbonden met
    Joomla's database;

    ontvangt en bewaart
    MAC adressen

Externe Joomla Website

Maatwerk Component
beheer van
    collega's +
    MAC adressen

toont
    aanwezigen...

Daarom webservice:

Mobile App (Android)
    maakt verbinding met
    webservice

    toont alle collega's
    en hun aanwezigheid

Beetje theorie

HTTP request...

een record:
GET - 1 of meer opvragen
POST – record aanmaken
PUT – update een record
DELETE – verwijderen

... op Endpoints

meervoud: /members/
ipv /member/

URL: /members/1
ipv /members.php?id=1

GEEN ww! maar woord + http request: Toevoegen: /members/ + POST
ipv /members/add

Dus: CRUD

Create - POST /members
Read - GET /members/1
Update - PUT /members/1
Delete - DELETE /members/1

List - GET /members

Employees -
GET /companies/X/members

Output: XML of JSON?

XML document formaat:
* tekst georienteerd
* niet positie georienteerd
maar wel: tags, tags, tags, tags, tags, tags, tags, tags...
en closing tags: tags, tags, tags, tags, tags, tags, tags...

JSON: data formaat
* veel lichter dan XML
* whitespace buiten quotes -> ignored

Output: JSON formaat

Verschillende stijlen:
* JSON-API
* Twitter API
* Facebook API

JSON - JSON-API Stijl

{
"data": [{
	"type": "employee",
	"id": "1",
	"attributes": { "name": "Maurice Moss" }
	"relationships": {
	"company": {
		"links": {
			"self": "http://example.com/employee/1/relationships/company",
			"related": "http://example.com/company/1"
		},
		"data": { "type": "company", "id": "1" }
		}
	}
  }
]
}

JSON - Twitter Stijl

[
	{
		"id": "1",
		"name": "Maurice Moss",
		"company": "Reynholm Industries",
	},
	{
		"id": "2",
		"name": "Roy Trenneman",
		"company": "Reynholm Industries",
	}
]

JSON - Facebook Stijl

{
	"data": [
		{
			"id": "1",
			"name": "Maurice Moss",
			"company": "Reynholm Industries",
		},
		{
			"id": "2",
			"name": "Roy Trenneman",
			"company": "Reynholm Industries",
		}
	]
}

Whoisthere: JSON

{msg: [
	{	id: "1",
		name: "Maurice Moss",
		company: "Reynholm Industries",
		photo: "images/itcrowd/moss.jpg",
		datetime: null
	},
	{	id: "2",
		name: "Roy Trenneman",
		company: "Reynholm Industries",
		photo: "images/itcrowd/roy.jpg",
		datetime: "2017-04-02 15:50:04"
	} ],
error: false,
status: 200
}

Status codes

HTTP Status codes:
2xx - success
3xx - redirection
4xx - client errors
5xx - server errors

Eigen error codes

JSON-API errors

{
"errors": {
		"code" : "1337",
		"title": "Elite level needed",
		"details": "Obviously you didn't played the game in the 1980s.",
		"href": "http://example.com/docs/errors/1337"
	}
}

Debugging

Browser:
Postman (Chrome Add-on)
HttpFox (FireFox Add-on)

Network:
Wireshark
Fiddler
Charles (betaald)

Versioning

URI of hostname:
http://example.com/api/v1/
http://v1.example.com/api/

HTTP body / URL query
{ "version" : "1.0" }

Request Header
APIversion: 1.0
Vary: APIversion

Content Negociation
Accept: application/vnd.myapp.v1+json

Whoisthere: Versioning

Geen versioning

1e versie nog niet verder doorontwikkeld.

Authentication

Basic:
Username + Password

Digest:
MD5 hash

OAuth 1.0a
Access token exchange
public key + secret key

OAuth 2.0
SSL verplicht!
via URL of HTTP Request header

Whoisthere: Authentication

Security through obscurity

Pagination

Beperk grootte
HTTP response

Query string
/members?limit=10

Mijn webservice: Pagination

Geen pagination:
(nog) klein aantal medewerkers

Documentation

Wat documenteren?
* API Endpoints
* Voorbeeld code
* Tutorial

Tools:
* API Blueprint
* RAML
* Swagger

Whoisthere: Documentation

/api/

{
	msg: "Members API v1. README.md contains API information.",
	error: false,
	status: 200
}

/api/README.md

# Hubbels API v1
This application outputs JSON data. Use
- /members/ to display all members
- /members/:id to display a specific member
- /connections/ to display all connected members

### Version
3.0.2

Linked Data

JSON-LD
http://json-ld.org
* "Microdata" voor JSON
* definieert context

http://schema.org/ bevat ook JSON-LD voorbeelden.

Linked Data voorbeeld

{
  "@context": "http://json-ld.org/contexts/person.jsonld",
  "@id": "http://dbpedia.org/resource/John_Lennon",
  "name": "John Lennon",
  "born": "1940-10-09",
  "spouse": "http://dbpedia.org/resource/Cynthia_Lennon"
}

Linked Data voorbeeld

{"@context":
	{
		"Person": "http://xmlns.com/foaf/0.1/Person",
		"xsd": "http://www.w3.org/2001/XMLSchema#",
		"name": "http://xmlns.com/foaf/0.1/name",
		"nickname": "http://xmlns.com/foaf/0.1/nick",
		"affiliation": "http://schema.org/affiliation",
		"depiction":
		{
			"@id": "http://xmlns.com/foaf/0.1/depiction",
			"@type": "@id"
		},
		"image":
		{
			"@id": "http://xmlns.com/foaf/0.1/img",
			"@type": "@id"
		},

Open Data

Kwaliteit Open Data
Ontsluiting data:

1. vrij toegankelijk, in welk formaat dan ook.
2. in gestructureerd formaat (bijv Excel).
3. in Open Formaat (bijv CSV)
4. via een RESTful JSON API
5. via RESTful JSON-LD API
6. in het RDF (Resource Description Framework) formaat

Bron: Zes Sterren Model (Dimitri van Hees)

Whoisthere

Whoisthere bash script

#!/bin/bash
# scan all MAC addresses in local network and send to external server
# (c) 2015 Peter Martin, db8.nl

MACS=$(sudo arp-scan --interface=eth0 --localnet -quiet | grep -E -o 'xdigit:{2}(:xdigit:{2}){5}')

array=(${MACS// / })
for i in "${!array[@]}"
do
MAC+="{\"mac\":\"${array[i]}\"},"
done

MAC="{\"macs\":["${MAC::-1}"]}"

curl -X POST -H "Content-Type: application/json" -d $MAC -i http://example.com/api/mac/

Whoisthere API script 1/6

initialise();

require 'Slim/Slim.php';

\Slim\Slim::registerAutoloader();

Whoisthere API script 2/6

$app = new \Slim\Slim(array(
	'mode' => 'development'
));

$app->db = JFactory::getDbo();
$app->view(new \JsonApiView());
$app->add(new \JsonApiMiddleware());

//Main entry
$app->get('/', function () use ($app) {
	$app->render(200, array(
		'msg' => 'Members API v1. README.md contains API information.',
	));
});

Whoisthere API script 3/6

/* Members */
$app->map('/members/', function () use ($app) {
	$query = $app->db->getQuery(true)
	->select('#__db8members_members.id AS id, name, cat.title AS company, photo, datetime')
	->from($app->db->quoteName('#__db8members_members'))
	->leftjoin('#__db8members_devices ON #__db8members_devices.member_id = #__db8members_members.id')
	->leftjoin('#__db8members_connections ON #__db8members_connections.mac = #__db8members_devices.mac')
	->leftjoin('#__categories AS cat ON cat.id = #__db8members_members.catid')
	->where('#__db8members_members.state = 1')
	->order('#__db8members_members.name')
	->group('#__db8members_members.id');
	$app->db->setQuery($query);
	$results = $app->db->loadObjectList();
	$app->render(200, array( 'msg' => $results, ));
})->via('GET');

Whoisthere API script 4/6

/* Member */
$app->map('/members/:id', function ($id) use ($app) {
	$query = $app->db->getQuery(true)
    ->select('name, company, position, website')
    ->from($app->db->quoteName('#__db8members_members'))
    ->where('id = ' . $id)
    ->where('state = 1')

    $app->db->setQuery($query);
    $result = $app->db->loadObject();
    $app->render(200, array(
        'msg' => $result,
    ));
})->via('GET');

Whoisthere API script 5/6

/* Connections */
$app->map('/mac/', function () use ($app) {
	$request = $app->request();
	$body = $request->getBody();
	$macs = json_decode($body)->macs;
	$results="";

	//delete all records
	$app->db->setQuery('TRUNCATE TABLE #__db8members_connections');
	$app->db->execute();

	//insert all mac addresses
	$query = $app->db->getQuery(true);
	$columns = array('id', 'mac', 'datetime');
	$values = array();
	$now =JFactory::getDate();

Whoisthere API script 6/6

	foreach ($macs as $mac)
	{
		$values[] = '0, "'.$mac->mac.'", "'.$now.'"';
	}
	$query->insert($app->db->quoteName('#__db8members_connections'));
	$query->columns($columns);
	$query->values($values);
	$app->db->setQuery($query);
	$app->db->query();

	$app->render(200, array(
		'msg' => $macs, //results,
	));

})->via('POST');
$app->run();

Thank you!

Presentation:
http://slides.db8.nl

Peter Martin
e-mail: info at db8.nl
twitter: @pe7er
presentation: https://db8.nl

Sources

* API Evangelist
* Build APIs You Won't Hate - Philip Sturgeon
* REST API implementaties - Bèr Kessels

* http://json-ld.org
* Platform Linked Data Nederland
* Linked Data in HTML

Photo Credits 1/2

Cat Sleeping Garden Domestic Brown Tabby Cute - Ranya, 2007
Cat Face Sleep Exhausted White Cat Pet - congerdesign, 2016
Digitization Null One Binary System Binary Digital - Gerd Altmann, 2017
The Sandwich
REST API in WordPress 4.4
Slim framework
Digging in permafrost - Nick Bonzey, 2007
Giving a gift - Yolanda DeLoach, 2015
Cables Wires, The Combination Of Connect - Michal Jarmoluk, 2012
CRUD Operations - Priya Darshini, 2013
Merge-split-transwiki default 2 - David Gothberg, 2008
Papertape - Poil, 2005

Photo Credits 2/2

USS Kennedy - status board update - Mate Airman Casey Jones, 2002
Bending - Holotone, 2007
Continuous Versioning Strategy for Internal APIs - Bill Doerrfeld, 2017
Authenticate
Do Ostriches Really Bury Their Heads in the Sand? - wonderopolis
Text document with page number icon - Benjamin D. Esham, 2007
Ray en Anita (2 Unlimited) - ANP, 2015
No Limits (record sleeve) - 2 Unlimited
Bonanjo - Marta Pucciarelli, 2013
Glass ochem - Purpy Pupple, 2010
Confused Muddled Illogical Disoriented - Steve Buissinne, 2015
All other photos: Peter Martin