Quick Start Guide

Table of Contents

• YouTube Video
• Who is Funnel for?
• What is Funnel?
• Why use Funnel?
• How do I use Funnel?
  • Step 1: Create a Funnel User Account
  • Step 2: Define an Avro Schema Representing Your Data
  • Step 3: Obtain an Access Token for your User Account
  • Step 4: Add/Update your Avro Schema in Funnel
  • Step 5: Upload Data for a Schema
  • Step 6: Query for Data
• Congratulations

YouTube Video

Who is Funnel for?

The Funnel system is built to serve companies that deal with high volumes of incoming data and updates and who must also serve high volumes of queries on such data, and who are looking to lower their costs and/or improve their performance.

Common industries where the Funnel system can be used include:

• Ride Hailing: Taxi services and Bike/Scooter Rentals
• Event Management: Guidance and coordination for tours, summits, and events.
• Security: Coordination of security guards or military personnel.
• Traffic Control: Shipping, Drones

What is Funnel?

Funnel is a software system capable of supporting high volumes of data updates as well as efficient arbitrary queries on such data. It presents a secured Application Program Interface (API) which can be used to define and update data formats, to send data, and to query data. The data formats of its APIs are easy to use and are based on Apache Avro and JSON.

Why use Funnel?

Funnel is written in a highly efficient manner which gives it high performance and low costs, and its software architecture makes trade-offs to more efficiently support the handling of large volumes of real-time data. Using Funnel can be considered a cost-saving measure, because it gives companies the benefits of a full software engineering team working on a custom solution for 6 months, for approximately 1/40th the cost.

How do I use Funnel?

Step 1: Create a Funnel User Account

Navigate to https://api.funnel-labs.io/web/index.html, click the “Register” link, and follow the sequence to create a new account. This account is in the “Free Tier” and will be rate-limited, thus is not suitable for production use. Upgrades can later be arranged by contacting accounts@funnel-labs.io and describing your needs.

The account contains several important pieces of information:

• A distinct username and email.
  In this guide, we will assume the username is “ExampleInc“.
• Credentials to authorize access.
  In this guide, we will assume the password is “abcd1234“.
• One or more “namespaces” which you will own, and under which your data will be organized.
  In this guide, we will assume the namespace is “com.example“.

Step 2: Define an Avro Schema Representing Your Data

In order to upload data, one must first define the data format that will be used. Additionally, the fields that will be searchable need to be designated, so that indexes can be created by the Funnel system to optimize the execution of queries.

Apache Avro Schemas are an open source standard, and their format is further described in the Funnel docs.

For the sake of this guide, we will define the following schema which will reside in the file named “staff.avsc”.

{
  "namespace": "com.example",
  "name": "Staff",
  "type": "record",
  "fields": [
    {
      "name": "id",
      "type": "int",
      "idx_primary": true
    },
    {
      "name": "job",
      "type": {
        "type": "enum",
        "name": "Job",
        "symbols": ["COOK", "WAITER", "SECURITY"]
      },
      "idx_type": "equality"
    },
    {
      "name": "loc",
      "type": {
        "type": "record",
        "name": "Location",
        "fields": [
          {"name": "latitude", "type": "double"},
          {"name": "longitude", "type": "double"}
        ]
      },
      "idx_type": "spatial"
    },
    {
      "name": "rating",
      "type": "double",
      "idx_type": "comparing"
    },
    {
      "name": "name",
      "type": "string",
      "idx_type": "comparing"
    }
  ]
}
Code language: JSON / JSON with Comments (json)

The schema above defines a data type called “Staff” which resides in the namespace “com.example“, which is owned by the user account “ExampleInc”. This data record has a few items of interest:

• The field “id” is an integer, it uniquely identifies Staff, and may be searched using equality operations in queries, e.g. “id = 3”.
• The field “job” can take on one of three different values: COOK, WAITER, or SECURITY.
• The field “loc” represents a (latitude, longitude) coordinate, and may be searched using “within” clauses, e.g. “loc within 1.5 km of (13.3, 52.1)“.
• The field “rating” can have decimal values, and it can be used for comparison queries, e.g. “rating > 3.0“.
• The field “name” can have string values like “Bob”, and it can also be used for comparison queries like “name = "Bob"” or “name < "Sam"“.

Step 3: Obtain an Access Token for your User Account

Most interactions with the Funnel system will be performed by back-end software systems in your company, which will obtain short-lived “access tokens” to keep your data secure. These Access Tokens, one obtained, can be repeatedly used over a short period of time to authenticate requests to resources owned by your User Account.

(In this example, the curl tool will be used to make HTTP request, but any other tool may be used.)

Make a request for Access and Refresh Tokens using the following command:

$ curl -s -X POST https://auth.funnel-labs.io/auth/realms/funnel/protocol/openid-connect/token -d "client_id=service&grant_type=password&username=ExampleInc&password=abcd1234"
{
"access_token": "eyJh...RFA",
"expires_in": 300,
"refresh_expires_in": 1800,
"refresh_token": "eyJh...n2E",
"token_type": "Bearer",
"not-before-policy": 0,
"session_state": "f821e5f1-e6c5-4392-a5ac-162d79e00c45",
"scope": "funnel-mouth funnel-spout"
}

For the purposes of running additional commands via the command line, it is often easier to save the value of the Access Token into a shell environment variable. This can be done using a small script to pick out the Access Token like so:

$ ACCESS_TOKEN=$(curl -s -X POST https://auth.funnel-labs.io/auth/realms/funnel/protocol/openid-connect/token -d "client_id=service&grant_type=password&username=ExampleInc&password=abcd1234" | jq -r '.access_token')

Note: The credentials should be kept secret, and never made available to your customers or other unauthorized parties. Back-end systems should retrieve these values using secret-management tools.

Step 4: Add/Update your Avro Schema in Funnel

The Access Token obtained above can be used to authenticate an update to schemas in the namespace owned by the User Account.

In this example, the “@” symbol is used to insert the contents of a file, namely the schema file defined in Step 2.

$ curl -D - https://api.funnel-labs.io/admin/schemas \
  -X POST -d '@staff.avsc' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $ACCESS_TOKEN"

OK
Code language: JavaScript (javascript)

It can be confirmed that the schema file has been uploaded by reading it back from the system, using its namespace and name as part of the URL.

$ curl https://api.funnel-labs.io/admin/schemas/com.example/Staff \
  -H "Authorization: Bearer $ACCESS_TOKEN" | jq

{
  "name": "Staff",
  "type": "record",
  "fields": [
    // ...
  ],
  "namespace": "com.example"
}
Code language: JavaScript (javascript)

Great, the new schema is loaded and ready to use. A single user account may have many schemas, each of which can be independently updated, upload data, or query data.

Note: Schemas are stored permanently and not lost over time.

Step 5: Upload Data for a Schema

With a schema established, data can be uploaded either from your back-end systems or directly from your customer’s mobile apps, which have been forwarded and Access Token for this purpose. In the following example, the curl command will be used to upload a data record that is in a format consistent with the schema above.

$ curl https://api.funnel-labs.io/users/ExampleInc/data/com.example/Staff \
  -X POST -d '
  {"id": 3,
   "job": "WAITER",
   "loc": {"latitude": 1.0, "longitude": 2.0},
   "rating": 1.3,
   "name": "Bob"}' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $ACCESS_TOKEN"

OK
Code language: PHP (php)

Note: Funnel focuses primarily on current data, thus old data may eventually expire.

Note: It is important to submit data fields in the same order in which they appear in the schema.

Step 6: Query for Data

The data that is uploaded may be queried using the Funnel Query Language using queries such as “rating > 2.0 and loc within 500m of (13.3, 52.2)” or “job = WAITER or name = "Bob"“.

$ curl 'https://api.funnel-labs.io/users/ExampleInc/data/com.example/Staff?query=name="Bob"' \
  -H "Authorization: Bearer $ACCESS_TOKEN"

[
  {
    "id": 3,
    "job": "WAITER",
    "loc": {"latitude": 1, "longitude": 2},
    "rating": 1.3,
    "name": "Bob"
  }
]
Code language: JavaScript (javascript)

The queries may be initiated by your business back-end systems or directly from your customer’s Mobile App if Access Tokens are distributed to them from your back-end systems.

Congratulations

At this point, you have examples of how to perform all the basic operations using the Funnel system, including account creation, schema definition, data uploading, and data querying. For more detailed information, see the Funnel Docs.