Data Streaming

In this guide, we will look at how to setup and use Enterprise Data Streaming (or "EDS" for short).

The Data Streaming functionality is only available with Enterprise Subscriptions. Please contact Sales if you would like more information about using EDS as part of your integration.

Background

EDS provides the ability for you to receive semi real-time database change data capture events from the Shopmonkey database for your data. These changes are automatically collected and partitioned for your data and you can stream this data to the EDS server automatically. The EDS server is a free, open source server provided by Shopmonkey which you can use to stream this data to your own database.

Architecture

eds

Our network runs the Shopmonkey infrastructure and the software that your shop uses when you access our software. You are responsible for running an instance of the EDS server in your network and under your control. The EDS server can run anywhere that can reach the Internet (a cloud provider, your own data center, your corporate IT network, etc). All communication between your EDS server and our network are encrypted using TLS. The EDS server can only communicate to our network and read data from your organization based on a private credential that you will be provided.

The EDS server will pull changes automatically when running. Database changes should typically arrive within 200ms or less. However, the latency depends on many factors including the latency between your EDS server and one of our cloud endpoints, your connection to your database and the performance of the database you're using, etc. Currently, the EDS service is not rate limited. However, we reserve the right to add rate limiting if we see specific issues which impact the overall quality of service.

Database Support

The following databases are currently supported:

  • PostgreSQL
  • CockroachDB
  • Microsoft SQL Server
  • Snowflake

Coming soon:

  • MySQL

However, we will accept Pull Requests for additional databases. If you need assistance, please contact us.

Running the Server

Prerequisites

Before you can run the EDS server you will need to obtain a credentials file from Shopmonkey. This file is a private key and should be treated as a password and stored securely. If you lose your key or your key is compromised, please contact us as soon as possible to revoke this credential. Anyone gaining access to this credential will be able to read all of your data from our system.

Download

You can download the latest version from the Releases page or you can build the binary yourself (see the README in the repo).

Run the Server

You can start the server using the following command:

./eds-server server --creds [CREDS] [URL]

Replace [URL] with the connection string for your database. For example, for a local PostgreSQL database you might use postgresql://root@localhost:26257/test?sslmode=disable.

Replace [CREDS] with the appropriate values. The [CREDS] value should be the path to your credentials file provided to you, and the file name of the creds file should end in .creds.

The server will continue to run consuming data and creating records in your database until it is stopped. If you stop the server and restart it, it will pick up from where it left off.

In a production scenario, you must keep running your server constantly. If your server isn't running for more than 24 hours you experience data loss. However, server restarts within less than 24 hour periods will continue processing once restarted.

Optional Flags

--verbose

  • Description: Turn on verbose logging.

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --verbose

--silent

  • Description: Turn off all logging.

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --silent

--trace-nats

  • Description: Turn on lower-level NATS tracing.

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --trace-nats

--dump-dir

  • Description: Write each incoming message to the specified directory.

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --dump-dir <directory_path>

--dry-run

  • Description: Simulate loading without actually making database changes.

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --dry-run

--consumer-prefix

  • Description: Add a consumer group prefix to the name.

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --consumer-prefix <prefix_name>

--timestamp

  • Description: Add timestamps to logging.
  • Usage: 
    ./eds-server server --creds [CREDS] [URL] --timestamp

--importer

  • Description: Mass-migrate data from your Shopmonkey instance to your external database. This can be used to initially seed your external database, or import missing data from a period of time. A pre-signed URL containing a ZIP folder of the imported data can be generated to start this process. Please reach out to a Shopmonkey representative for assistance in generating the pre-signed URL.
  • Usage:
./eds-server server --creds [CREDS] [URL] --importer <unzipped folder directory>

--consumer-start-time

  • Description: Creates a NATS consumer that EDS will read from to re-process old messages up to 1 week old into your external database.
  • Usage:
./eds-server server --creds [CREDS] [URL] --consumer-start-time <1h - 168h>

--ignore-local-nats

  • Description: This disables re-publishing of NATS messages and has your database provider read from the main NATS server

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --ignore-local-nats

--port

  • Description: The port to run the local NATS server on. Configuring this is mainly useful if you are running multiple EDS instances on the same machine.

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --port 4224

--health-port

  • Description: The port to run the health check server on. Configuring this is mainly useful if you are running multiple EDS instances on the same machine.

  • Usage:

    ./eds-server server --creds [CREDS] [URL] --port 8081

Security

Transport

When using EDS, the communication is one-way from our network to your network and all data is transmitted fully encrypted end-to-end.

PII

Personally Identifiable Information (or "PII" for short) data in the database is tokenized. For example, email addresses will not be provided in clear text and will instead have an opaque token such as 1234-5678-90ab-cdef-ghij. For any PII fields that you would like to decode, you must use the API for that endpoint to decode that data directly. The reason is that we do not store PII data in our database and use a third-party secure data vault to protect your data and the data for your customers.

For example, to decode the email for a Customer from the database, you can use the id from the record and use the Find Customer API to fetch the emails in plain text. When using the API, the data is automatically decoded from the data vault on-the-fly.

Development Environment Setup Demo

Here's a quick video to show how to get your local development environment setup:

Data Export and Import Process

Documentation for Data Export and Import Process

This documentation outlines the process for exporting and importing data through a series of HTTP requests and command-line operations. The process involves four main steps: starting the export, generating a pre-signed URL, downloading the file, and importing the file into your database.

Step 1: Start the Export

To initiate the data export, send a POST request to the /v3/export endpoint. This request should include a list of tables you wish to export in the request body. Below are a few examples that you can replace with the tables you need. Request:

Example Request

POST
/v3/export
{
	"tables": ["appointment", "label", "timesheet"]
}

Response:

{
	"success": true,
	"data": {
		"fileName": "EXAMPLE_FILE_NAME"
	}
}

Step 2: Generate the Pre-Signed URL

After receiving the filename from the export process, the next step is to generate a pre-signed URL to download the exported file. Send a POST request to the /v3/export/presigned_url endpoint with the filename. Request:

Example Request

POST
/v3/export/presigned_url
{
	"fileName": "EXAMPLE_FILE_NAME"
}

Response:

{
	"success": true,
	"data": {
		"url": "$PRESIGNED_GCP_URL"
	}
}

Step 3: Download the File

Use the pre-signed URL obtained in the previous step to download the file. You can do this with a command-line tool like curl or in any programming language that supports HTTP requests. Example Command:

curl -o "./downloaded_file.gz" "$PRESIGNED_GCP_URL"

Step 4: Import the File

Once the file is downloaded, you can import the data into your database using the provided importer tool. Below is the full command to run the importer. Command:

./eds-server server --creds ../path/to/creds/shopmonkey_shop.creds --consumer-prefix MyEDSConsumer --importer "." "$DB_URL"

Parameters:

  • --creds: Path to the credentials file needed to authenticate the database connection.
  • --consumer-prefix: A prefix to identify the consumer of the data. This is used to maintian position on the stream.
  • --importer: Path to the folder containing the Gzipped JSON Lines files from Step 3

For assistance in obtaining a pre-signed URL to download the Gzipped JSON Lines files or any other questions, please reach out to your representative at Shopmonkey.

Available Tables List

Appointments

  • appointment
  • appointment_user_connection
  • reminder

Authorization

  • authorization
  • authorization_service

Canned Services

  • canned_service
  • canned_service_fee
  • canned_service_labor
  • canned_service_part
  • canned_service_subcontract
  • canned_service_tire

Customers and Vehicles

  • customer
  • customer_location_connection
  • mileage_log
  • referral_source
  • service
  • vehicle
  • vehicle_location
  • vehicle_owner

Inspections

  • inspection
  • inspection_item
  • inspection_item_quick_note
  • inspection_template
  • inspection_template_item
  • tpi_scan_inspection
  • tpi_scan

Inventory

  • brand
  • core_charge
  • inventory_category
  • inventory_fee
  • inventory_labor
  • inventory_part
  • inventory_tire
  • part
  • tire_model
  • tire_pressure_log
  • tire
  • vendor

Labels

  • label
  • saved_label

Labors

  • labor_matrix_range
  • labor_matrix
  • labor_rate
  • labor

Messaging

  • conversation
  • email
  • message
  • message_template
  • message_thread_metadata
  • phone_number_opt_in
  • phone_number

Orders

  • order
  • order_signed_invoice
  • work_request
  • workflow_status

Purchase Orders

  • purchase_order_fee
  • purchase_order_part
  • purchase_order_tire
  • purchase_order

Settings

  • fee
  • latest_payment_receipt
  • location
  • payment
  • payment_term
  • pricing_matrix_range
  • pricing_matrix
  • statement
  • subcontract
  • tax_config
  • user

Time Logs

  • timesheet