Data sync providers¶
Warning
This feature is considered experimental. It might change at any time without prior notice.
pretix provides connectivity to many external services through plugins. A common requirement
is unidirectionally sending (order, customer, ticket, …) data into external systems.
The transfer is usually triggered by signals provided by pretix core (e.g. order_placed
),
but performed asynchronously.
Such plugins should use the OutboundSyncProvider
API to utilize the queueing, retry and mapping mechanisms as well as the user interface for configuration and monitoring.
An OutboundSyncProvider
for registering event participants in a mailing list could start
like this, for example:
1from pretix.base.datasync.datasync import OutboundSyncProvider
2
3class MyListSyncProvider(OutboundSyncProvider):
4 identifier = "my_list"
5 display_name = "My Mailing List Service"
6 # ...c
The plugin must register listeners in signals.py for all signals that should to trigger a sync and
within it has to call MyListSyncProvider.enqueue_order()
to enqueue the order for synchronization:
@receiver(order_placed, dispatch_uid="mylist_order_placed")
def on_order_placed(sender, order, **kwargs):
MyListSyncProvider.enqueue_order(order, "order_placed")
Furthermore, most of these plugins need to translate data from some pretix objects (e.g. orders) into an external system’s data structures. Sometimes, there is only one reasonable way or the plugin author makes an opinionated decision what information from which objects should be transferred into which data structures in the external system.
Otherwise, you can use a PropertyMappingFormSet
to let the user set up a mapping from pretix model fields
to external data fields. You could store the mapping information either in the event settings, or in a separate
data model. Your implementation of OutboundSyncProvider.mappings
needs to provide a list of mappings, which can be e.g. static objects or model instances, as long as they
have at least the properties defined in
pretix.base.datasync.datasync.StaticMapping
.
1# class MyListSyncProvider, contd.
2 def mappings(self):
3 return [
4 StaticMapping(
5 id=1, pretix_model='Order', external_object_type='Contact',
6 pretix_id_field='email', external_id_field='email',
7 property_mappings=self.event.settings.mylist_order_mapping,
8 ))
9 ]
Currently, we support orders and order positions as data sources, with the data fields defined in
pretix.base.datasync.sourcefields.get_data_fields()
.
To perform the actual sync, implement sync_object_with_properties()
and optionally
finalize_sync_order()
. The former is called for each object to be created according to the mappings
.
For each order that was enqueued using enqueue_order()
:
each Mapping with
pretix_model == "Order"
results in one call tosync_object_with_properties()
,each Mapping with
pretix_model == "OrderPosition"
results in one call tosync_object_with_properties()
per order position,finalize_sync_order()
is called one time after all calls tosync_object_with_properties()
.
Implementation examples¶
For example implementations, see the test cases in tests.base.test_datasync
.
In SimpleOrderSync
, a basic data transfer of order data only is
shown. Therein, a sync_object_with_properties
method is defined as follows:
1from pretix.base.datasync.utils import assign_properties
2
3def sync_object_with_properties(
4 self, external_id_field, id_value, properties: list, inputs: dict,
5 mapping, mapped_objects: dict, **kwargs,
6):
7 # First, we query the external service if our object-to-sync already exists there.
8 # This is necessary to make sure our method is idempotent, i.e. handles already synced
9 # data gracefully.
10 pre_existing_object = self.fake_api_client.retrieve_object(
11 mapping.external_object_type,
12 external_id_field,
13 id_value
14 )
15
16 # We use the helper function ``assign_properties`` to update a pre-existing object.
17 update_values = assign_properties(
18 new_values=properties,
19 old_values=pre_existing_object or {},
20 is_new=pre_existing_object is None,
21 list_sep=";",
22 )
23
24 # Then we can send our new data to the external service. The specifics of course depends
25 # on your API, e.g. you may need to use different endpoints for creating or updating an
26 # object, or pass the identifier separately instead of in the same dictionary as the
27 # other properties.
28 result = self.fake_api_client.create_or_update_object(mapping.external_object_type, {
29 **update_values,
30 external_id_field: id_value,
31 "_id": pre_existing_object and pre_existing_object.get("_id"),
32 })
33
34 # Finally, return a dictionary containing at least `object_type`, `external_id_field`,
35 # `id_value`, `external_link_href`, and `external_link_display_name` keys.
36 # Further keys may be provided for your internal use. This dictionary is provided
37 # in following calls in the ``mapped_objects`` dict, to allow creating associations
38 # to this object.
39 return {
40 "object_type": mapping.external_object_type,
41 "external_id_field": external_id_field,
42 "id_value": id_value,
43 "external_link_href": f"https://example.org/external-system/{mapping.external_object_type}/{id_value}/",
44 "external_link_display_name": f"Contact #{id_value} - Jane Doe",
45 "my_result": result,
46 }
Note
The result dictionaries of earlier invocations of sync_object_with_properties()
are
only provided in subsequent calls of the same sync run, such that a mapping can
refer to e.g. the external id of an object created by a preceding mapping.
However, the result dictionaries are currently not provided across runs. This will
likely change in a future revision of this API, to allow easier integration of external
systems that do not allow retrieving/updating data by a pretix-provided key.
mapped_objects
is a dictionary of lists of dictionaries. The keys to the dictionary are
the mapping identifiers (mapping.id
), the lists contain the result dictionaries returned
by sync_object_with_properties()
.
In OrderAndTicketAssociationSync
, an example is given where orders, order positions,
and the association between them are transferred.
The OutboundSyncProvider base class¶
- class pretix.base.datasync.datasync.OutboundSyncProvider(event)¶
- close()¶
Called after all orders of an event have been synced. Can be used for clean-up tasks (e.g. closing a session).
- classmethod enqueue_order(order, triggered_by, not_before=None)¶
Adds an order to the sync queue. May only be called on derived classes which define an
identifier
attribute.Should be called in the appropriate signal receivers, e.g.:
@receiver(order_placed, dispatch_uid="mysync_order_placed") def on_order_placed(sender, order, **kwargs): MySyncProvider.enqueue_order(order, "order_placed")
- Parameters:
order – the Order that should be synced
triggered_by – the reason why the order should be synced, e.g. name of the signal (currently only used internally for logging)
- filter_mapped_objects(mapped_objects, inputs)¶
For order positions, only
- finalize_sync_order(order)¶
Called after
sync_object
has been called successfully for all objects of a specific order. Can be used for saving bulk information per order.
- property mappings¶
Implementations must override this property to provide the data mappings as a list of objects.
They can return instances of the
StaticMapping
namedtuple defined above, or create their own class (e.g. a Django model).- Returns:
The returned objects must have at least the following properties:
id: Unique identifier for this mapping. If the mappings are Django models, the database primary key should be used. This may be referenced in other mappings, to establish relations between objects.
pretix_model: Which pretix model to use as data source in this mapping. Possible values are the keys of
sourcefields.AVAILABLE_MODELS
external_object_type: Destination object type in the target system. opaque string of maximum 128 characters.
pretix_id_field: Which pretix data field should be used to identify the mapped object. Any
DataFieldInfo.key
returned bysourcefields.get_data_fields()
for the combination ofEvent
andpretix_model
.external_id_field: Destination identifier field in the target system.
property_mappings: Mapping configuration as generated by
PropertyMappingFormSet.to_property_mappings_list()
.
- next_retry_date(sq)¶
Optionally override to configure a different retry backoff behavior
- should_sync_order(order)¶
Optionally override this method to exclude certain orders from sync by returning
False
- sync_object_with_properties(external_id_field: str, id_value, properties: list, inputs: dict, mapping: ObjectMapping, mapped_objects: dict, **kwargs) dict | None ¶
This method is called for each object that needs to be created/updated in the external system – which these are is determined by the implementation of the mapping property.
- Parameters:
external_id_field – Identifier field in the external system as provided in
mapping.external_identifier
id_value – Identifier contents as retrieved from the property specified by
mapping.pretix_identifier
of the model specified bymapping.pretix_model
properties – All properties defined in
mapping.property_mappings
, as list of three-tuples(external_field, value, overwrite)
inputs – All pretix model instances from which data can be retrieved for this mapping. Dictionary mapping from sourcefields.ORDER_POSITION, .ORDER, .EVENT, .EVENT_OR_SUBEVENT to the relevant Django model. Most providers don’t need to use this parameter directly, as properties and id_value already contain the values as evaluated from the available inputs.
mapping – The mapping object as returned by
self.mappings
mapped_objects – Information about objects that were synced in the same sync run, by mapping definitions before the current one in order of
self.mappings
. Type is a dictionary{mapping.id: [list of OrderSyncResult objects]}
Useful to create associations between objects in the target system.
Example code to create return value:
1return { 2 # optional: 3 "action": "nothing_to_do", # to inform that no action was taken, because the data was already up-to-date. 4 # other values for action (e.g. create, update) currently have no special 5 # meaning, but are visible for debugging purposes to admins. 6 7 # optional: 8 "external_link_href": "https://external-system.example.com/backend/link/to/contact/123/", 9 "external_link_display_name": "Contact #123 - Jane Doe", 10 "...optionally further values you need in mapped_objects for association": 123456789, 11}
The return value needs to be a JSON serializable dict, or None.
Return None only in case you decide this object should not be synced at all in this mapping. Do not return None in case the object is already up-to-date in the target system (return “action”: “nothing_to_do” instead).
This method needs to be idempotent, i.e. calling it multiple times with the same input values should create only a single object in the target system.
Subsequent calls with the same mapping and id_value should update the existing object, instead of creating a new one. In a SQL database, you might use an INSERT OR UPDATE or UPSERT statement; many REST APIs provide an equivalent API call.
- sync_queued_orders(queued_orders)¶
This method should catch all Exceptions and handle them appropriately. It should never throw an Exception, as that may block the entire queue.
Property mapping format¶
To allow the user to configure property mappings, you can use the PropertyMappingFormSet,
which will generate the required property_mappings
value automatically. If you need
to specify the property mappings programmatically, you can refer to the description below
on their format.
- class pretix.control.forms.mapping.PropertyMappingFormSet(pretix_fields, external_fields, available_modes, prefix, *args, **kwargs)¶
A simple JSON-serialized property_mappings
list for mapping some order information can look like this:
1[
2 {
3 "pretix_field": "email",
4 "external_field": "orderemail",
5 "value_map": "",
6 "overwrite": "overwrite",
7 },
8 {
9 "pretix_field": "order_status",
10 "external_field": "status",
11 "value_map": "{\"n\": \"pending\", \"p\": \"paid\", \"e\": \"expired\", \"c\": \"canceled\", \"r\": \"refunded\"}",
12 "overwrite": "overwrite",
13 },
14 {
15 "pretix_field": "order_total",
16 "external_field": "total",
17 "value_map": "",
18 "overwrite": "overwrite",
19 }
20]
Translating mappings on Event copy¶
Property mappings can contain references to event-specific primary keys. Therefore, plugins must register to the event_copy_data signal and call translate_property_mappings on all property mappings they store.
- pretix.base.datasync.utils.translate_property_mappings(property_mappings, checkin_list_map)¶
To properly handle copied events, users of data fields as provided by get_data_fields need to register to the event_copy_data signal and translate all stored references to those fields using this method.
For example, if you store your mappings in a custom Django model with a ForeignKey to Event:
1@receiver(signal=event_copy_data, dispatch_uid="my_sync_event_copy_data") 2def event_copy_data_receiver(sender, other, checkin_list_map, **kwargs): 3 object_mappings = other.my_object_mappings.all() 4 object_mapping_map = {} 5 for om in object_mappings: 6 om = copy.copy(om) 7 object_mapping_map[om.pk] = om 8 om.pk = None 9 om.event = sender 10 om.property_mappings = translate_property_mappings(om.property_mappings, checkin_list_map) 11 om.save()