Siaqodb can be synchronized automatically with the SiaqodbCloud module to a server side NoSQL database like CouchDB or MongoDB via a RESTful service called SiaqodbCloud Service. SiaqodbCloud Service is an ASP.NET WebAPI project and it is provided as Open Source on Github, so you can customize it, add support for another database engine and host it in your preferred cloud or your own data center.

Note: SiaqodbCloud also offers support for synchronization with Azure Storage click here to read more.

Only Buckets from Siaqodb's DocumentStore can be synchronized with server side databases.

In order to synchronize clients with a master database, you will need three main components:

  • Siaqodb-powered client application (Clients)
  • Siaqodb-Cloud Web Service (Server)
  • CouchDB or MongoDB Database
Here is a layout of how the tool works:
Siaqodb Cloud Network Map

Server Database Configuration

MongoDB
MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need. MongoDB stores data in JSON-like documents.
You can download MongoDB from here (min version 3.2.x), install it and then create a config file (mongo.cfg) that looks something like this:

   systemlog
       destination file
       path c:\data\log\mongod.log
   storage
       dbpath c:\data\db
   replication
       oplogsizemb 100
       replsetname rs0
Start MongoDB with this config (for example on windows "C:\mongodb\bin\mongod.exe" --config "C:\mongodb\mongo.cfg" ). The important section in the above config is the 'replication' section which will enable MongoDB to store all changes in oplog.rs collection. the entire sync process relies on Mongo's oplog. After MongoDB server is started, start mongo shell and initialize replica set on 'local' database:
> use local
> rs.initiate()
now the collection 'oplog.rs' should be created on 'local' database.

Finally create a database called 'siaqodb'. (If you prefer another name, you can change it in src\Repository\MongoDB\MongoDBRepo.cs). Then, inside the database created, create 2 collections: 'sys_accesskeys' and 'sys_synclog'. (The names can be changed also in \src\Repository\CouchDB\MongoDBRepo.cs ).

CouchDB
Apache CouchDB (not to be confused with Couchbase Server) is open source database software that focuses on ease of use and having an architecture that "completely embraces the Web". It has a document-oriented NoSQL database architecture and is implemented in the concurrency-oriented language Erlang; it uses JSON to store data, JavaScript as its query language using MapReduce, and HTTP for an API.
You can download CouchDB from here (min version 1.6.1), install it and then create 2 databases: 'sys_accesskeys' and 'sys_synclog'. (The names can be changed in \src\Repository\CouchDB\CouchDBRepo.cs).
CouchDB uses HTTP protocol and the database URL is set in the Siaqodb-Cloud Service project. If you are not using the default CouchDB port (or if your CouchDB is hosted on another server), you will need to edit \src\Repository\CouchDB\CouchDBRepo.cs and modify the default CouchDB URL:
private const string DbServerUrl = "http://127.0.0.1:5984/";
with your own database connection URL.

SiaqodbCloud Service

SiaqodbCloud service is Open Source and it can be downloaded from Github. It is an ASP.NET WebAPI project and after you have successfully installed and started the project, it can be used as a replication end point for Siaqodb. By default, MongoDB and CouchDB are supported as NoSQL databases for synchronization. To synchronize Siaqodb's Buckets, you will need to create a database in CouchDB (or a collection in a MongoDB) for each Siaqodb Bucket. For instance, if you have in Siaqodb a bucket called "invoices" (accessed by: siaqodb.Documents["invoices"]), you will have to create a collection called 'invoices' in a MongoDB database or a database 'invoices' in CouchDB, so all documents from the bucket 'invoices' will be synchronized with MongoDB/CouchDB documents from database/collection 'invoices'.

Project Setup
Download and open SiaqodbCloud-Service in Visual Studio. This project is the RESTful web service that Siaqodb clients will use to synchronize with your server database. Once your project is ready for production, this service will need to be deployed to a server in the cloud. First, you will need to change the repository type in the code to match the database you are using. You can choose the backend database (MongoDB or CouchDB) by modifying Repository/RepositoryFactory.cs:

public static IRepository GetRepository()
{
    return new MongoDB.MongoDBRepo();
    //return new CouchDB.CouchDBRepo();
}

Next you may need to adjust the port in which your web service will run for debugging (default port is 10000). In Visual Studio click Project > SiaqodbCloudService Properties > Web > Configure your Project Url. For instance: http://localhost:11735/ (ensure that the port is open and not already used by another project or program).
This port number can be changed if you desire, but should match in both the client and cloud-service projects. You may also need to adjust the port in your client code when you deploy your web service to a production port like 80 or 443.

Note: for testing, you may find the need to run the project on a separate host using IIS Express. IIS Express is a lightweight, self-contained version of IIS optimized for developers and runs your web service project in Visual Studio. By default, IIS Express will not allow remote connections. Please refer to this post on Stack Overflow to enable remote connections for IIS Express.

Service Authentication

Requests to SiaqodbCloud service are signed with a HMAC-SHA256 signature. Client code needs an 'access_key_id' (which is public and included in the header of the request) and a 'secret_key' which is used to build HMAC-SHA256 signature. The 'secret_key' must be provided in the client app but is never transmitted. 'access_key_id' and 'secret_key' should be stored in sys_accesskeys database/collection.

Open up the data editor for your database; for example, with CouchDB, simply navigate to the following URL:

http://localhost:5984/_utils/
You need to manually create a database called 'sys_accesskeys', then add one single record to the database that contains one data item: 'secretkey'. The _id will be generated for you automatically in the database and you will need that _id in your client code. Here is an example of a JSON record stored in CouchDB database or MongoDB collection called 'sys_accesskeys':
{
    "_id": "3ba69b5835dgdb308766b4756b00079a",
    "secretkey": "4362kljh63k4599hhgm"
}
secretkey is just a string that you made up; think of it as a password for your service call. The _id field will be needed for the client calls in the next section.

SiaqodbCloud Client

Any Bucket from Siaqodb can be synchronized via SiaqodbCloud only if the bucket is marked as Sync-able. To mark it, you will have to call:

 Sqo.SiaqodbConfigurator.SetSyncableBucket("invoices", true);

To start sync-ing buckets, you will have to reference SiaqodbCloud.dll assembly which can be found inside the Siaqodb package. After you add the reference, create a SiaqodbSync instance by providing:

  • URL to the Siaqodb Cloud Web API service.
  • AccessKeyId which will be sent in the header of every HTTP request. This is the '_id' field from the record you inserted into your sys_accesskeys database.
  • SecretKey which will be used to build Signature that signs every HTTP request. This is the 'secretkey' field from the record you inserted into your sys_accesskeys database.

Example:

using SiaqodbCloud;
.....
SiaqodbSync syncContext = new SiaqodbSync("http://localhost:11735/v0/",  // host and port running the Siaqodb-Cloud Service
    "3ba69b5835dgdb308766b4756b00079a",   // _id from your database record in your sys_accesskeys database
    "4362kljh63k4599hhgm");  // secretkey field from the record you inserted into your sys_accesskeys database

You can synchronize a Siaqodb local Bucket with an online Bucket by using Push/Upload and Pull/Download operations.

Upload all changes from an offline bucket to a cloud bucket:


IBucket bucket = siaqodb.Documents["invoices"];
syncContext.Push(bucket);//push local changes to server

Download all changes from the cloud bucket:

IBucket bucket = siaqodb.Documents["invoices"];
syncContext.Pull(bucket);//pull server changes to client db

Important note: every Pull() operation will call Push() first therefore Pull() will provide a complete Sync

Filtering

It does not always make sense to download all data from a cloud bucket to the local device. SiaqodbCloud allows you to filter and download only changes of records by a certain criteria, example:

 IBucket bucket = siaqodb.Documents["invoices"];
Filter f = new Filter("year");
f.Value = 2016;
syncContext.Pull(bucket,f); // pull server changes to client but only for filtered records

Conflicts

When an app runs with poor connections, and many concurrent users may do modifications on data, or even the same user using different devices, clearly conflicts may appear. SiaqodbSync can handle those conflicts and let developer decide based on business criteria, how to merge or what record is the "winner" of a conflict.

Conflicts are detected based on Document.Version property. This property is updated by server only at every UPDATE/DELETE/INSERT operation. The client code should never update this property.

Following types of conflicts may appear:

Delete - Delete: a record is deleted locally by one user, but the same record is deleted online by another user. When the synchronization is made, at push, a conflict appears;

Delete - Update: a record is deleted locally by one user, but the same record is updated online by another user. When the synchronization is made, at push, a conflict appears;

Update - Delete: a record is locally updated by one user, but the same record has been deleted online by another user. When the synchronization is made, at push, a conflict appears;

Update - Update: a record is locally updated by one user, but the same record is updated online by another user. When the synchronization is made, at push, a conflict appears;

By default, the online record is the 'winner', so for example on an update-update conflict, after a Push() operation occurs, the local Document changes are rejected and the server version is downloaded and stored locally.

However this behavior may be overridden and you may merge both conflicted documents or choose as winner based on app business criteria. To override these rules, you have to pass an instance of a class that implements 'IConflictResolver' to the Push/Pull methods. Example:

 private class LocalResolver : IConflictResolver
{
    public Document Resolve(Document local, Document online)
    {
        return local;
    }
}

And now you can pass an instance of this class to Pull/Push methods:

syncContext.Pull(bucket, new LocalResolver());//also calls Push()