Data Serialization

Relevant source files

Purpose and Scope

The data serialization system provides unified, type-aware conversion of arbitrary Python objects into JSON-serializable formats for API responses, component outputs, and data persistence. It handles truncation of large values, conversion of complex types (Pydantic models, pandas DataFrames, LangChain documents, numpy arrays), and graceful handling of unserializable objects.

For information about how components use serialization to format their output artifacts, see Component Lifecycle. For API-level data exchange and response formatting, see API Endpoints.

Serialization Architecture

The serialization system uses a dispatcher pattern to route objects to type-specific handlers, with recursive processing for nested structures and configurable truncation limits.

Serialization Flow

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L12-L18" min=12 max=18 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L189-L251" min=189 max=251 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L253-L306" min=253 max=306 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Type Dispatcher Pattern

The _serialize_dispatcher() function uses pattern matching to route objects to specialized handlers based on their type. It handles standard library types, third-party library types (LangChain, Pandas, Numpy), and Langflow-specific entities.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L189-L251" min=189 max=251 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Type-Specific Handlers

Primitive Types

The system handles Python primitives without conversion, with specific checks for invalid float values.

Type	Handler	Output
`None`	`_serialize_primitive`	`None`
`int`	`_serialize_primitive`	Integer value
`float`	`_serialize_primitive`	Float value (or `None` if NaN/Inf)
`bool`	`_serialize_primitive`	Boolean value
`complex`	`_serialize_primitive`	Complex value

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L121-L130" min=121 max=130 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

String and Byte Types

Strings and bytes support truncation to prevent oversized API responses:

Type	Handler	Truncation Logic
`str`	`_serialize_str()`	If `len(obj) > max_length`: `obj[:max_length] + "..."`
`bytes`	`_serialize_bytes()`	Decode to UTF-8 (ignoring errors), then apply string truncation

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L43-L66" min=43 max=66 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Temporal and Numeric Types

Standard conversions for common Python types:

Type	Handler	Conversion
`datetime`	`_serialize_datetime()`	Convert to UTC ISO format: `obj.replace(tzinfo=timezone.utc).isoformat()`
`Decimal`	`_serialize_decimal()`	Convert to `float`
`UUID`	`_serialize_uuid()`	Convert to `str`

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L69-L81" min=69 max=81 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Pydantic Models

The system supports both modern Pydantic (v2) and legacy (v1) models, ensuring compatibility with various LangChain versions and custom components.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L94-L105" min=94 max=105 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Collection Types

Collections are processed recursively with optional truncation to manage memory footprint.

Dictionary Serialization:

List/Tuple Serialization:

If len(obj) > max_items: truncate to max_items elements and append "... [truncated N items]"
Otherwise: recursively serialize each item

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L107-L119" min=107 max=119 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Pandas and Numpy Types

Pandas structures and Numpy types are converted to standard Python types (lists, dicts, primitives) to ensure JSON compatibility.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L146-L161" min=146 max=161 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef> <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L168-L187" min=168 max=187 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Truncation Strategies

The serialization system implements truncation to prevent memory issues and oversized responses. These limits are typically retrieved from the system settings.

Text Truncation

Controlled by max_length parameter (defaults to MAX_TEXT_LENGTH from settings):

Configuration:

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L31-L34" min=31 max=34 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef> <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L43-L55" min=43 max=55 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Collection Truncation

Controlled by max_items parameter (defaults to MAX_ITEMS_LENGTH from settings):

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L37-L40" min=37 max=40 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef> <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L112-L118" min=112 max=118 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Sensitive Data Masking

The serialization layer includes specialized logic for TransactionTable and TransactionReadResponse to mask sensitive credentials before they are persisted or returned via the API.

Masking Logic

The system identifies sensitive keys (e.g., api_key, password, token) using a predefined set SENSITIVE_KEY_NAMES and a regex SENSITIVE_KEYS_PATTERN.

Partial Masking: If a value is longer than 12 characters (MIN_LENGTH_FOR_PARTIAL_MASK), it shows the first 4 and last 4 characters (e.g., sk-o...abcd).
Full Masking: Values shorter than the limit or non-string values are replaced with ***REDACTED***.
Exclusion: Specific keys like code are entirely excluded from logs.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/services/database/models/transactions/model.py#L14-L58" min=14 max=58 file-path="src/backend/base/langflow/services/database/models/transactions/model.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/services/database/models/transactions/model.py#L144-L155" min=144 max=155 file-path="src/backend/base/langflow/services/database/models/transactions/model.py">Hii</FileRef>

Sentinel Values and Error Handling

Unserializable Sentinel

When serialization fails or an object is encountered that cannot be converted to a JSON-compatible format, the system uses a sentinel value instead of raising exceptions to maintain flow stability.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/serialization/serialization.py#L23-L28" min=23 max=28 file-path="src/backend/base/langflow/serialization/serialization.py">Hii</FileRef>

Artifact Post-Processing

The post_process_raw function handles the final conversion of build results into frontend-ready data. It uses jsonable_encoder with CUSTOM_ENCODERS as a secondary fallback.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/schema/artifact.py#L65-L82" min=65 max=82 file-path="src/backend/base/langflow/schema/artifact.py">Hii</FileRef>

Integration Points

Data and JSON Classes

Langflow utilizes unified base types for data exchange. JSON is the modern base type, while Data is maintained as an alias for backward compatibility. Both utilize serialize_data for their internal state management.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/schema/data.py#L1-L9" min=1 max=9 file-path="src/backend/base/langflow/schema/data.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/schema/schema.py#L1-L24" min=1 max=24 file-path="src/backend/base/langflow/schema/schema.py">Hii</FileRef>

Database Model Serialization

SQLModel/Pydantic models for Flow, VertexBuildTable, and TransactionTable use field_serializer decorators to enforce serialization rules during database I/O.

VertexBuildTable: Serializes data, artifacts, and params fields using serialize() with enforced limits. <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/services/database/models/vertex_builds/model.py#L41-L66" min=41 max=66 file-path="src/backend/base/langflow/services/database/models/vertex_builds/model.py">Hii</FileRef>
Flow: Custom validators and serializers for icon, endpoint_name, and data ensure structural integrity before persistence. <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/services/database/models/flow/model.py#L117-L187" min=117 max=187 file-path="src/backend/base/langflow/services/database/models/flow/model.py">Hii</FileRef>
MessageTable: Implements serialize_timestamp to ensure UTC-normalized strings are stored. <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/services/database/models/message/model.py#L47-L58" min=47 max=58 file-path="src/backend/base/langflow/services/database/models/message/model.py">Hii</FileRef>

Message and Memory Serialization

The Message schema acts as the primary container for chat data, importing core definitions from lfx.schema.message. When stored in MessageTable, the from_message class method handles the conversion of complex fields like content_blocks and properties into JSON-serializable formats.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/schema/message.py#L7-L14" min=7 max=14 file-path="src/backend/base/langflow/schema/message.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/services/database/models/message/model.py#L73-L177" min=73 max=177 file-path="src/backend/base/langflow/services/database/models/message/model.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/memory.py#L141-L151" min=141 max=151 file-path="src/backend/base/langflow/memory.py">Hii</FileRef>

Data to Text Conversion

The data_to_text_list and safe_convert functions provide high-level utilities to serialize complex objects into formatted strings. safe_convert often uses orjson for JSON formatting with indentation for readability in logs or the Chat UI.

Function	Input	Serialization Logic
`safe_convert`	`Data`, `Message`, `DataFrame`	Custom logic per type, e.g., `to_markdown` for DataFrames.
`_serialize_data`	`Data`	`orjson.dumps` with `OPT_INDENT_2` wrapped in Markdown blocks.
`data_to_text`	`list[Data]`	Template-based formatting of serialized data.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/helpers/data.py#L108-L121" min=108 max=121 file-path="src/backend/base/langflow/helpers/data.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/helpers/data.py#L156-L189" min=156 max=189 file-path="src/backend/base/langflow/helpers/data.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/helpers/data.py#L26-L105" min=26 max=105 file-path="src/backend/base/langflow/helpers/data.py">Hii</FileRef>

Schema and Type Mapping

The system maps internal types to their serializable equivalents via get_type and build_output_logs. These ensure that LogType, OutputType, and InputType are consistently represented across the lfx and langflow packages.

Sources: <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/schema/schema.py#L1-L24" min=1 max=24 file-path="src/backend/base/langflow/schema/schema.py">Hii</FileRef>, <FileRef file-url="https://github.com/langflow-ai/langflow/blob/873d5585/src/backend/base/langflow/schema/artifact.py#L25-L52" min=25 max=52 file-path="src/backend/base/langflow/schema/artifact.py">Hii</FileRef>