XM Discover API: Export API Recommendations
About Export API Recommendations
Speed up your Export API implementation with these tips and suggestions:
- Requesting fields vs. attributes
- Use scroll ID to export large volumes
- Build breakpoints into your requests
- Download new or updated data only
- Reconsider daily exports from volatile projects
Requesting Fields vs. Attributes
If this is your first time using Export API, please request all data you need as attributes and don't look back! If you are already requesting fields and looking to transition, use the table below to accommodate the change in response names.
We made this change to reduce confusion: what we historically described as a "field" or an "attribute" was academic, and required you to remember the arbitrary difference when constructing your call. The transition to the simpler paradigm requires one-time attention and effort. Note that if you do not request any fields you will still receive a few items such as sentence ID and natural ID.
| Attributes (New) | Fields (Old) |
|---|---|
| _id_sentence | sentence_id |
| _id_verbatim | verbatim_id |
| _id_document | document_id |
| _id_source | source |
| _verbatimtype | verbatim_type |
| _doc_time | document_date |
| _id_batch | session_id |
| natural_id | natural_id |
| cb_content_type | content_type |
| cb_content_subtype | content_subtype |
| _degreesentimentindex | sentiment |
| _words | sentence |
| _language | processed_language |
| _languagedetected | autodetected_language |
| _sentence_start_pos | sentence_start_pos |
| _sentence_end_pos | sentence_end_pos |
| cb_date_updated | cb_date_updated |
| cb_date_created_utc | created_date |
Use Scroll ID to Export Large Volumes
Leave scroll ID empty for the first page of requested results. Scroll ID is used to retrieve subsequent pages of results. Each response includes a potentially unique scroll_id value that must be updated and included in the next requested page.
Scroll ID is valid for 2 minutes after generation of a response, so you will need to ensure followup requests occur within that time. When crafting your usage of the Export API, experiment with different variations of page size and scrolling behavior to optimize timing and throughput.
Build Breakpoints into Your Requests
Due to the distributed nature of how XM Discover stores data there is no guarantee that the order of records in exports will be consistent; the same request will produce the same records in total, but they may be in a different order. The result is that restarting an export "in the middle" of a sequence of scrolls is not possible.
We highly recommend segmenting export requests by a well-understood attribute such as date. Instead of requesting a month of records in a single request, break up the request into smaller segments using a filter, for example for 1 day, week, or even a few hours of data at a time. This makes it less painful in the event of an interruption: instead of restarting your entire export from scratch you can restart only the impacted subset.
Download New or Updated Data Only
Avoid re-exporting unchanged data by filtering requests by cb_date_updated and cb_date_created_utc. The cb_date_created_utc field stores when the record was first received by XM Discover. As the name implies, cb_date_updated stores the last timestamp that the given sentence or document was updated. With these two you can easily filter to export only newly loaded data or data updated since your last export.
This can greatly reduce the amount of data you export to help keep you within your plan and speed up your export. In the Export API cb_date_updated and cb_date_created_utc are exposed by using the parameters created_before/created_since and updated_before/updated_since.
Reconsider Daily Exports from Volatile Projects
Projects may update frequently. For example, reclassifying a project may change categorization for all or large swaths of records. If you attempt to export all "updated" data daily, this may result in having to re-export large amounts of data on a daily basis. After discussions with many customers we've found that daily exports of all updated data is often overkill: weekly or monthly exports to re-sync XM Discover data to your exported data is more than sufficient.
This is particularly true if there is no one in your business waiting to use daily data. Instead of downloading a frequently updated record multiple times through the week, consider waiting until the end of the week to re-export the record once. This can reduce the amount of data you export to help keep you within your plan and reduce maintenance.