Incremental Load

Incremental data loading, reading changed data

Introduction

Once an initial data load has been executed, it is more optimal to only read what has been changed since the last data load. This results in faster loading times, reduced stress on the server and the consumer, and less bandwidth usage. Overall, it is more cost effective. Additionally, less energy is consumed :green_heart:

Flow

The Enterprise Documents returned by the ObjectAPI always have the complex property lastUpdated, providing information on when this document is last modified and by whom.

[  
    {
        "additionalInformation": {
            "sequenceNumber": 9
        },
        "companyId": "U4Demo",
        "lastUpdated": {
            "updatedAt": "2025-04-25T10:41:26.860Z",
            "updatedBy": "214163"
        },
        "transactionNumber": 1500048427
    }
]```

This can be used in incremental load scenarios, providing the start moment of the previous (full/incremental) data load.

```bash
/v2/objects/general-ledger-transactions? \
    &companyId=u4Demo \
    &posted=true \
    &filter=lastUpdated/updatedAt ge 2025-04-25T23:33:01.053Z \
    &orderBy=companyId,transactionNumber,additionalInformation/sequenceNumber

The suggestion is to use the ge operator to make sure no changes are lost at the edge. This means that even though we do not lose data at the edge, we might end up receiving the same Enterprise Documents (duplicate).

Additionally, if many changes are expected to occur between two data loading cycles (full or incremental), pagination techniques need to be applied as previously described.