mex.common.models.base package

Submodules

mex.common.models.base.container module

class mex.common.models.base.container.ItemsContainer(*, items: list[_ContainerItemT])

Bases: BaseModel, Generic[_ContainerItemT]

Generic container that contains items.

items: list[_ContainerItemT]

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'items': FieldInfo(annotation=list[~_ContainerItemT], required=True)}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class mex.common.models.base.container.PaginatedItemsContainer(*, items: list[_ContainerItemT], total: int)

Bases: BaseModel, Generic[_ContainerItemT]

Generic container that contains items and has a total item count.

items: list[_ContainerItemT]

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'items': FieldInfo(annotation=list[~_ContainerItemT], required=True), 'total': FieldInfo(annotation=int, required=True)}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

total: int

mex.common.models.base.extracted_data module

class mex.common.models.base.extracted_data.ExtractedData(*, hadPrimarySource: MergedPrimarySourceIdentifier, identifierInPrimarySource: Annotated[str, MinLen(min_length=1), MaxLen(max_length=1000), _PydanticGeneralMetadata(pattern='^[^\\n\\r]+$')])

Bases: BaseModel

Base model for all extracted item classes.

This class adds two important attributes for metadata provenance: hadPrimarySource and identifierInPrimarySource, which are used to uniquely identify an item in its original primary source. The attribute stableTargetId has to be set by each concrete subclass, like ExtractedPerson, because it needs to have the correct type, e.g. MergedPersonIdentifier.

This class also adds a validator to automatically set identifiers for provenance.

_get_identifier(identifier_type: type[_ExtractedIdentifierT]) → _ExtractedIdentifierT

Consult the identity provider to get the identifier for this item.

Parameters:

identifier_type – ExtractedIdentifier-subclass to cast the identifier to

Returns:

Identifier of the correct type

_get_stable_target_id(identifier_type: type[_MergedIdentifierT]) → _MergedIdentifierT

Consult the identity provider to get the stableTargetId for this item.

Parameters:

identifier_type – MergedIdentifier-subclass to cast the identifier to

Returns:

StableTargetId of the correct type

hadPrimarySource: Annotated[MergedPrimarySourceIdentifier, FieldInfo(annotation=NoneType, required=True, description='The stableTargetId of the primary source, that this item was extracted from. This field is mandatory for all extracted items to aid with data provenance. Extracted primary sources also have this field and are all extracted from a static primary source for MEx. The extracted primary source for MEx has its own merged item as a primary source.', frozen=True)]

identifierInPrimarySource: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='This is the identifier the original item had in its source system. It is only unique amongst items coming from the same system, because identifier formats are likely to overlap between systems. The value for `identifierInPrimarySource` is therefore only unique in composition with `hadPrimarySource`. MEx uses this composite key to assign a stable and globally unique `identifier` per extracted item.', examples=['123456', 'item-501', 'D7/x4/zz.final3'], frozen=True, metadata=[MinLen(min_length=1), MaxLen(max_length=1000), _PydanticGeneralMetadata(pattern='^[^\\n\\r]+$')])]

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'populate_by_name': True, 'str_max_length': 100000, 'str_min_length': 1, 'str_strip_whitespace': True, 'use_enum_values': True, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'hadPrimarySource': FieldInfo(annotation=MergedPrimarySourceIdentifier, required=True, description='The stableTargetId of the primary source, that this item was extracted from. This field is mandatory for all extracted items to aid with data provenance. Extracted primary sources also have this field and are all extracted from a static primary source for MEx. The extracted primary source for MEx has its own merged item as a primary source.', frozen=True), 'identifierInPrimarySource': FieldInfo(annotation=str, required=True, description='This is the identifier the original item had in its source system. It is only unique amongst items coming from the same system, because identifier formats are likely to overlap between systems. The value for `identifierInPrimarySource` is therefore only unique in composition with `hadPrimarySource`. MEx uses this composite key to assign a stable and globally unique `identifier` per extracted item.', examples=['123456', 'item-501', 'D7/x4/zz.final3'], frozen=True, metadata=[MinLen(min_length=1), MaxLen(max_length=1000), _PydanticGeneralMetadata(pattern='^[^\\n\\r]+$')])}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

mex.common.models.base.filter module

class mex.common.models.base.filter.BaseFilter

Bases: BaseModel

Base class for filter implementations.

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class mex.common.models.base.filter.FilterField(*, fieldInPrimarySource: str | None = None, locationInPrimarySource: str | None = None, examplesInPrimarySource: list[str] | None = None, filterRules: Annotated[list[FilterRule], MinLen(min_length=1)], comment: str | None = None)

Bases: BaseModel

Filter definition for one field in the primary source.

comment: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, title='comment')]

examplesInPrimarySource: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, title='examplesInPrimarySource')]

fieldInPrimarySource: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, title='fieldInPrimarySource')]

filterRules: Annotated[list[FilterRule], FieldInfo(annotation=NoneType, required=True, title='filterRules', metadata=[MinLen(min_length=1)])]

locationInPrimarySource: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, title='locationInPrimarySource')]

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'comment': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, title='comment'), 'examplesInPrimarySource': FieldInfo(annotation=Union[list[str], NoneType], required=False, default=None, title='examplesInPrimarySource'), 'fieldInPrimarySource': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, title='fieldInPrimarySource'), 'filterRules': FieldInfo(annotation=list[FilterRule], required=True, title='filterRules', metadata=[MinLen(min_length=1)]), 'locationInPrimarySource': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, title='locationInPrimarySource')}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class mex.common.models.base.filter.FilterRule(*, forValues: list[str] | None = None, rule: str | None = None)

Bases: BaseModel

A single filter rule to apply.

forValues: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, title='forValues')]

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'forValues': FieldInfo(annotation=Union[list[str], NoneType], required=False, default=None, title='forValues'), 'rule': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, title='rule')}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

rule: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, title='rule')]

mex.common.models.base.mapping module

class mex.common.models.base.mapping.BaseMapping

Bases: BaseModel

Base class for mapping implementations.

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class mex.common.models.base.mapping.MappingField(*, fieldInPrimarySource: str | None = None, locationInPrimarySource: str | None = None, examplesInPrimarySource: list[str] | None = None, mappingRules: Annotated[list[MappingRule[_MappingRuleT]], MinLen(min_length=1)], comment: str | None = None)

Bases: BaseModel, Generic[_MappingRuleT]

Generic mapping field model.

comment: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, title='comment')]

examplesInPrimarySource: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, title='examplesInPrimarySource')]

fieldInPrimarySource: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, title='fieldInPrimarySource')]

locationInPrimarySource: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, title='locationInPrimarySource')]

mappingRules: Annotated[list[MappingRule[_MappingRuleT]], FieldInfo(annotation=NoneType, required=True, title='mappingRules', metadata=[MinLen(min_length=1)])]

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'comment': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, title='comment'), 'examplesInPrimarySource': FieldInfo(annotation=Union[list[str], NoneType], required=False, default=None, title='examplesInPrimarySource'), 'fieldInPrimarySource': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, title='fieldInPrimarySource'), 'locationInPrimarySource': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, title='locationInPrimarySource'), 'mappingRules': FieldInfo(annotation=list[MappingRule[~_MappingRuleT]], required=True, title='mappingRules', metadata=[MinLen(min_length=1)])}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class mex.common.models.base.mapping.MappingRule(*, forValues: list[str] | None = None, setValues: _ValueT | None = None, rule: str | None = None)

Bases: BaseModel, Generic[_ValueT]

Generic mapping rule model.

forValues: Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, title='forValues')]

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {'forValues': FieldInfo(annotation=Union[list[str], NoneType], required=False, default=None, title='forValues'), 'rule': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, title='rule'), 'setValues': FieldInfo(annotation=Union[~_ValueT, NoneType], required=False, default=None, title='setValues')}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

rule: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, title='rule')]

setValues: Annotated[_ValueT | None, FieldInfo(annotation=NoneType, required=True, title='setValues')]

mex.common.models.base.merged_item module

class mex.common.models.base.merged_item.MergedItem

Bases: BaseModel

Base model for all merged item classes.

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'populate_by_name': True, 'str_max_length': 100000, 'str_min_length': 1, 'str_strip_whitespace': True, 'use_enum_values': True, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

mex.common.models.base.model module

class mex.common.models.base.model.BaseModel

Bases: BaseModel

Common base class for all MEx model classes.

classmethod _convert_list_to_non_list(field_name: str, value: list[Any]) → Any

Convert a list value to a non-list value by unpacking it if possible.

classmethod _convert_non_list_to_list(field_name: str, value: Any) → list[Any] | None

Convert a non-list value to a list value by wrapping it in a list.

classmethod _fix_value_listyness_for_field(field_name: str, value: Any) → Any

Check actual and desired shape of a value and fix it if necessary.

checksum() → str

Calculate md5 checksum for this model.

Creates a deterministic hash of the model by serializing it to JSON with sorted keys and computing an MD5 digest.

Returns:

MD5 hexdigest string representing the model’s current state.

classmethod fix_listyness(data: Any, handler: ValidatorFunctionWrapHandler) → Any

Adjust the listyness of to-be-parsed data to match the desired shape.

If that data is a Mapping and the model defines a list[T] field but the raw data contains just a value of type T, it will be wrapped into a list. If the raw data contains a literal None, but the list field is defined as required, we substitute an empty list.

If the model does not expect a list, but the raw data contains a list with no entries, it will be substituted with None. If the raw data contains exactly one entry, then it will be unpacked from the list. If it contains more than one entry however, an error is raised, because we would not know which to choose.

Parameters:

• data – Raw data or instance to be parsed

• handler – Validator function wrap handler

Returns:

data with fixed list shapes

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'ignore', 'populate_by_name': True, 'str_max_length': 100000, 'str_min_length': 1, 'str_strip_whitespace': True, 'use_enum_values': True, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

classmethod model_json_schema(by_alias: bool = True, ref_template: str = '#/$defs/{model}', schema_generator: type[~pydantic.json_schema.GenerateJsonSchema] = <class 'mex.common.models.base.schema.JsonSchemaGenerator'>, mode: ~typing.Literal['validation', 'serialization'] = 'validation') → dict[str, Any]

Generates a JSON schema for a model class.

Parameters:

• by_alias – Whether to use attribute aliases or not.

• ref_template – The reference template.

• schema_generator – Overriding the logic used to generate the JSON schema

• mode – The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

classmethod verify_computed_field_consistency(data: Any, handler: ValidatorFunctionWrapHandler) → Any

Validate that parsed values for computed fields are consistent.

Parsing a dictionary with a value for a computed field that is consistent with what that field would have computed anyway is allowed. Omitting values for computed fields is perfectly valid as well. However, if the parsed value is different from the computed value, a validation error is raised.

Parameters:

• data – Raw data or instance to be parsed

• handler – Validator function wrap handler

Returns:

data with consistent computed fields.

mex.common.models.base.preview_item module

class mex.common.models.base.preview_item.PreviewItem

Bases: BaseModel

Base model for previews of merged items.

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'populate_by_name': True, 'str_max_length': 100000, 'str_min_length': 1, 'str_strip_whitespace': True, 'use_enum_values': True, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

mex.common.models.base.rules module

class mex.common.models.base.rules.AdditiveRule

Bases: BaseModel

Base rule to add values to merged items.

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'populate_by_name': True, 'str_max_length': 100000, 'str_min_length': 1, 'str_strip_whitespace': True, 'use_enum_values': True, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class mex.common.models.base.rules.PreventiveRule

Bases: BaseModel

Base rule to prevent primary sources for fields of merged items.

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'populate_by_name': True, 'str_max_length': 100000, 'str_min_length': 1, 'str_strip_whitespace': True, 'use_enum_values': True, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class mex.common.models.base.rules.RuleSet

Bases: BaseModel

Base class for a set of an additive, subtractive and preventive rule.

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'populate_by_name': True, 'str_max_length': 100000, 'str_min_length': 1, 'str_strip_whitespace': True, 'use_enum_values': True, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

class mex.common.models.base.rules.SubtractiveRule

Bases: BaseModel

Base rule to subtract values from merged items.

model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'populate_by_name': True, 'str_max_length': 100000, 'str_min_length': 1, 'str_strip_whitespace': True, 'use_enum_values': True, 'validate_assignment': True, 'validate_default': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, FieldInfo]] = {}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

mex.common.models.base.schema module

class mex.common.models.base.schema.JsonSchemaGenerator(by_alias: bool = True, ref_template: str = '#/$defs/{model}')

Bases: GenerateJsonSchema

Customization of the pydantic class for generating JSON schemas.

handle_ref_overrides(json_schema: Dict[str, Any]) → Dict[str, Any]

Disable pydantic behavior to wrap top-level $ref keys in an allOf.

For example, pydantic would convert

{“$ref”: “#/$defs/APIType”, “examples”: [“api-type-1”]}

into

{“allOf”: {“$ref”: “#/$defs/APIType”}, “examples”: [“api-type-1”]}

which is in fact recommended by JSON schema, but we need to disable this to stay compatible with mex-editor and mex-model.

Module contents