Civilisation Atlas | Automated Ingestion and Routing Protocol

article:
  id: CA-UI-007
  title: Automated Ingestion and Routing Protocol
  version: 1.0
  status: canonical
  type: specification
  parent_protocol: CA-UI-001
  depends_on:
    - CA-UI-002
    - CA-UI-003
    - CA-UI-004
    - CA-UI-005
    - CA-UI-006

purpose: >
  Define how AI receives an unclassified item, determines what
  it is, assigns provisional Atlas identities and coordinates,
  connects it to the knowledge graph, records provenance and
  routes it to the correct Civilisation Atlas locations.

INPUT
  ↓
IDENTIFY
  ↓
CLASSIFY
  ↓
LOCATE
  ↓
CONNECT
  ↓
VALIDATE
  ↓
ROUTE
  ↓
REGISTER

THE ATLAS DOES NOT REQUIRE
EVERY ARTICLE TO BE MANUALLY TAGGED.
THE SCHEMA EXISTS FIRST.
THE OBJECT ROUTES ITSELF
AGAINST THE SCHEMA.

routing_principle:
  manual_preclassification_required: false
  schema_required: true
  automated_inference_allowed: true
  provenance_required: true
  uncertainty_preserved: true
  human_review_available: true

input_types:
  - article
  - research_paper
  - book
  - historical_record
  - dataset
  - image
  - map
  - diagram
  - artefact_record
  - institution
  - technology
  - event
  - theory
  - claim
  - question
  - scenario
  - web_page
  - user_submission

ingestion_request:
  ingestion_id: required
  input_type: required
  input_reference: required
  submitted_at: required
  submitted_by: required
  declared_metadata:
    title: optional
    author: optional
    publication_date: optional
    language: optional
    source_type: optional
  requested_action:
    - classify
    - register
    - update
    - crosswalk
    - compare
    - answer
    - validate

processing_states:
  RECEIVED:
    definition: input_entered_runtime
  EXTRACTING:
    definition: structure_and_entities_being_detected
  IDENTIFIED:
    definition: candidate_objects_found
  CLASSIFIED:
    definition: object_types_and_scales_assigned
  LOCATED:
    definition: atlas_positions_assigned
  CONNECTED:
    definition: relationships_and_dependencies_generated
  VALIDATED:
    definition: schema_and_evidence_checks_completed
  ROUTED:
    definition: atlas_destinations_assigned
  REGISTERED:
    definition: object_committed_to_graph
  QUARANTINED:
    definition: unresolved_or_invalid_input_held_for_review
  REJECTED:
    definition: input_failed_minimum_acceptance_rules

extraction:
  entities:
    - people
    - organisations
    - institutions
    - civilisations
    - locations
    - technologies
    - infrastructures
    - events
    - theories
    - mechanisms
    - resources
    - dates
  statements:
    - observations
    - measurements
    - claims
    - causal_claims
    - comparisons
    - forecasts
    - normative_positions
  structures:
    - tables
    - figures
    - timelines
    - taxonomies
    - equations
    - references

function extract(input):
    content = parse(input)
    entities = detect_entities(content)
    claims = detect_claims(content)
    evidence = detect_evidence(content)
    relationships = detect_relationships(content)
    temporal_scope = detect_time(content)
    spatial_scope = detect_space(content)
    return extraction_bundle(
        entities,
        claims,
        evidence,
        relationships,
        temporal_scope,
        spatial_scope
    )

function identify_objects(bundle):
    candidates = []
    for entity in bundle.entities:
        candidate = {
            canonical_name:
                normalise_name(entity),
            aliases:
                detect_aliases(entity),
            object_type:
                infer_object_type(entity),
            temporal_scope:
                infer_temporal_scope(entity),
            spatial_scope:
                infer_spatial_scope(entity),
            system_level:
                infer_system_level(entity)
        }
        candidates.append(candidate)
    return candidates

identity_resolution:
  matching_fields:
    - canonical_name
    - aliases
    - external_identifiers
    - temporal_scope
    - spatial_scope
    - object_type
    - relationships
  outcomes:
    EXACT_MATCH:
      action: attach_to_existing_object
    PROBABLE_MATCH:
      action: attach_provisionally
    POSSIBLE_DUPLICATE:
      action: quarantine_for_resolution
    NEW_OBJECT:
      action: assign_new_canonical_id
    CONFLICTING_IDENTITY:
      action: preserve_competing_identity_records

function resolve_identity(candidate):
    matches = search_registry(candidate)
    if exact_match(matches):
        return existing_object_id
    if probable_match(matches):
        return provisional_link(matches.best)
    if possible_duplicate(matches):
        return quarantine(candidate)
    return create_object_id(candidate)

canonical_id:
  format: CA-[DOMAIN]-[OBJECT_TYPE]-[IDENTIFIER]
  domain_codes:
    HIS: historical
    PRE: present
    FUT: future
    EXT: external_research
    ECO: ecological
    TEC: technological
    INS: institutional
    INF: informational
    SYS: system
    GEO: geographic
    CIV: civilisation
  examples:
    - CA-HIS-CIV-ROMAN-EMPIRE
    - CA-TEC-INFRA-PRINTING-PRESS
    - CA-EXT-THEORY-KARDASHEV
    - CA-PRE-SYS-GLOBAL-FOOD-NETWORK

classification_dimensions:
  object_type: required
  system_level: required
  spatial_scope: required
  temporal_scope: required
  atlas_role: required

atlas_roles:
  - primary_civilisation_object
  - subsystem
  - capability_enabler
  - dependency_provider
  - control_node
  - information_source
  - theory_or_model
  - evidence_record
  - transition_driver
  - fracture_source
  - repair_mechanism
  - external_hazard
  - future_scenario

function classify(candidate):
    candidate.object_type =
        classify_by_schema(candidate)
    candidate.system_level =
        estimate_system_level(candidate)
    candidate.atlas_role =
        detect_operational_role(candidate)
    return candidate

function infer_system_level(object):
    if object.represents_humanity:
        return L8
    if object.represents_civilisation:
        return L7
    if object.represents_state:
        return L6
    if object.represents_city_or_region:
        return L5
    if object.represents_institution:
        return L4
    if object.represents_settlement:
        return L3
    if object.represents_local_group:
        return L2
    if object.represents_household:
        return L1
    return L0

NO COORDINATE ASSIGNMENT
BEFORE SCALE ASSIGNMENT.

coordinate_routing:
  direct:
    use_when: object_is_civilisation_scale
  inherited:
    use_when: object_belongs_to_larger_system
  modifying:
    use_when: object_changes_parent_coordinate
  cross_scale:
    use_when: object_affects_multiple_system_levels
  none:
    use_when: coordinate_not_applicable

function assign_atlas_position(object):
    if is_civilisation_scale(object):
        return direct_coordinate(
            capability = estimate_capability(object),
            lifecycle = estimate_lifecycle(object),
            habitat = estimate_habitat(object)
        )
    parent = locate_parent_system(object)
    effects = estimate_coordinate_effects(
        source = object,
        target = parent
    )
    return inherited_modifying_coordinate(
        inherited_from = parent,
        coordinate_effects = effects
    )

function estimate_capability(system):
    observed = detect_capability_bundle(system)
    for stage in descending(CAPABILITY_REGISTRY):
        if:
            reliable(stage, observed)
            and scaled(stage, observed)
            and institutionalised(stage, observed)
            and structurally_consequential(stage, observed):
            return stage
    return UNKNOWN

function estimate_lifecycle(system):
    indicators = collect(
        expansion,
        maintenance,
        legitimacy,
        institutional_health,
        resource_balance,
        repair_activity,
        capability_loss
    )
    phase = classify_phase(indicators)
    conditions = detect_conditions(
        drift,
        fracture,
        repair,
        transition,
        collapse
    )
    return {
        phase: phase,
        conditions: conditions
    }

function estimate_habitat(system):
    test = evaluate(
        local_survival,
        local_energy,
        local_food,
        local_water,
        local_maintenance,
        local_medicine,
        local_reproduction,
        local_governance,
        local_education,
        local_industrial_replacement
    )
    return highest_level_without_hidden_dependency(test)

civos_routes:
  - education
  - governance
  - production
  - distribution
  - memory
  - coordination
  - control
  - maintenance
  - repair
  - adaptation

function map_to_civos(object):
    mapped_mechanisms = []
    for mechanism in CIVOS_REGISTRY:
        if object_performs(object, mechanism):
            mapped_mechanisms.append(mechanism)
    return mapped_mechanisms

diagnostic_routes:
  fracture_map:
    trigger:
      - broken_flow
      - institutional_discontinuity
      - lower_floor_failure
      - capability_loss
  repair_map:
    trigger:
      - restoration_activity
      - substitution
      - reconnection
      - regeneration
  dependency_floor:
    trigger:
      - critical_requirement
      - single_point_of_failure
      - hidden_support_system
  control_geometry:
    trigger:
      - ownership
      - concentrated_authority
      - allocation_power
      - restricted_access
  sensor_and_signal_map:
    trigger:
      - measurement
      - monitoring
      - feedback
      - information_delay
  transition_test:
    trigger:
      - emerging_capability
      - institutional_adoption
      - stage_change_claim
  staircase_removal_test:
    trigger:
      - autonomy_claim
      - habitat_independence_claim
      - external_support_dependency

function route_diagnostics(object):
    destinations = []
    for diagnostic in DIAGNOSTIC_REGISTRY:
        if trigger_match(object, diagnostic):
            destinations.append(diagnostic.id)
    return destinations

function route_external_model(model):
    preserve(model.original_structure)
    crosswalk = map(
        objects = model.units_of_analysis,
        variables = model.variables,
        mechanisms = model.mechanisms,
        outcomes = model.outputs
    )
    classify_mapping(
        DIRECT,
        PARTIAL,
        COMPOSITE,
        DERIVED,
        ANALOGICAL,
        UNMAPPED,
        CONFLICTING
    )
    return crosswalk

EXTERNAL THEORY
DOES NOT BECOME ATLAS THEORY.
IT BECOMES
AN INTEROPERABLE MODEL
WITH AN ATLAS ADDRESS.

function generate_relationships(object, graph):
    candidates = semantic_match(
        object,
        graph.objects
    )
    edges = []
    for candidate in candidates:
        predicates =
            infer_valid_predicates(
                object,
                candidate
            )
        for predicate in predicates:
            edge = relationship(
                source = object.id,
                predicate = predicate,
                target = candidate.id,
                evidence = supporting_evidence(
                    object,
                    candidate,
                    predicate
                ),
                confidence = calculate_confidence()
            )
            edges.append(edge)
    return edges

function detect_dependencies(object):
    required_functions =
        identify_required_functions(object)
    dependencies = []
    for function in required_functions:
        providers =
            identify_function_providers(function)
        dependencies.append({
            required_function: function,
            providers: providers,
            criticality:
                estimate_criticality(function),
            substitutability:
                estimate_substitutability(providers),
            redundancy:
                count_independent_providers(providers),
            failure_effect:
                simulate_provider_failure(providers)
        })
    return dependencies

function attach_provenance(input, object):
    source_record =
        create_source_record(input)
    evidence_records =
        extract_evidence_records(input)
    claims =
        create_claim_records(
            evidence_records
        )
    provenance_chain =
        link(
            source_record,
            evidence_records,
            claims,
            object
        )
    return provenance_chain

routing_thresholds:
  AUTO_REGISTER:
    minimum_confidence: 0.90
    conditions:
      - identity_high
      - classification_high
      - evidence_traceable
      - no_critical_conflict
  PROVISIONAL_REGISTER:
    minimum_confidence: 0.70
    conditions:
      - identity_probable
      - schema_valid
      - uncertainty_explicit
  REVIEW_REQUIRED:
    minimum_confidence: 0.40
    conditions:
      - disputed_identity
      - weak_coordinate
      - conflicting_sources
      - major_schema_gap
  QUARANTINE:
    maximum_confidence: 0.39
    conditions:
      - insufficient_evidence
      - unresolved_duplicate
      - untraceable_claim
      - invalid_scale

provisional_record:
  provisional: true
  reason: required
  unresolved_fields: []
  confidence: required
  review_required: true
  expiry_or_review_date: required

UNCERTAINTY DOES NOT BLOCK INGESTION.
UNCERTAINTY CHANGES REGISTRATION STATUS.

function handle_ambiguity(candidate):
    interpretations =
        generate_candidate_classifications(candidate)
    preserve_all(
        interpretations_above_threshold
    )
    assign:
        primary_candidate
        alternative_candidates
        confidence_per_candidate
    if confidence_gap < required_margin:
        mark(REVIEW_REQUIRED)
    return candidate

classification_candidates:
  object: Byzantine Empire
  candidates:
    - type: civilisation
      confidence: 0.72
    - type: imperial_state
      confidence: 0.91
    - type: continuation_of_roman_state
      confidence: 0.84
  resolution:
    primary_type: imperial_state
    retained_secondary_classifications:
      - civilisation
      - Roman_continuity_system

function detect_duplicate(new_object):
    matches = graph_similarity_search(
        names,
        aliases,
        dates,
        locations,
        identifiers,
        relationships
    )
    if similarity >= 0.98:
        merge_as_same_identity()
    else if similarity >= 0.80:
        mark_possible_duplicate()
    else:
        register_as_new()

merge_rule:
  preserve_original_ids: true
  assign_canonical_surviving_id: true
  create_redirects: true
  merge_provenance: true
  erase_conflicting_data: false

function ingest_conflicting_claim(claim):
    existing = retrieve_claims_about_same_target(claim)
    if compatible(claim, existing):
        attach_as_corroboration()
    else:
        create_conflict_record(
            new_claim = claim,
            existing_claims = existing
        )
        preserve_all_positions()
        recalculate_confidence()
        mark_target_as_disputed()
    return

function detect_schema_gap(object):
    unmapped = find_unmapped_features(object)
    if unmapped is empty:
        return NO_GAP
    proposal = {
        gap_id: assign_gap_id(),
        source_object: object.id,
        unmapped_features: unmapped,
        affected_schemas: detect_affected_schemas(),
        proposed_action:
            "extend_or_crosswalk"
    }
    return proposal

FAILURE TO MAP MAY MEAN:
- BAD INPUT
- INSUFFICIENT EVIDENCE
- WRONG SCALE
- WRONG OBJECT TYPE
- OR AN ATLAS SCHEMA GAP

routing_destinations:
  object_registry:
    receives:
      - canonical_objects
      - aliases
      - classifications
  coordinate_registry:
    receives:
      - direct_coordinates
      - inherited_coordinates
      - coordinate_effects
  relationship_graph:
    receives:
      - structural_edges
      - causal_edges
      - temporal_edges
      - comparative_edges
  dependency_graph:
    receives:
      - dependencies
      - critical_paths
      - failure_propagations
  theory_crosswalk_registry:
    receives:
      - theories
      - disciplines
      - datasets
      - models
  evidence_registry:
    receives:
      - sources
      - claims
      - evidence
      - interpretations
      - provenance
  diagnostic_registry:
    receives:
      - fractures
      - repairs
      - control_geometry
      - signals
      - transitions
  control_tower:
    receives:
      - final_route_map
      - unresolved_items
      - conflicts
      - version_updates

route_manifest:
  ingestion_id: required
  object_id: required
  routes:
    object_registry: required
    coordinate_registry: optional
    relationship_graph: []
    dependency_graph: []
    theory_crosswalk_registry: optional
    evidence_registry: required
    diagnostic_registry: []
    article_ids: []
    external_depth_routes: []
  unresolved:
    fields: []
    conflicts: []
    schema_gaps: []
  registration_status: required
  confidence: required

route_manifest:
  ingestion_id: CA-INGEST-ROMAN-ROADS-001
  object_id: CA-HIS-INFRA-ROMAN-ROADS
  routes:
    object_registry: CA-UI-002
    coordinate_registry: CA-UI-003
    relationship_graph:
      - CA-UI-004
    evidence_registry:
      - CA-UI-006
    diagnostic_registry:
      - CA-MAP-DEPENDENCY
      - CA-MAP-CONTROL
      - CA-MAP-MAINTENANCE
    article_ids:
      - CA-ARTICLE-ROMAN-INFRASTRUCTURE
      - CA-ARTICLE-LOGISTICS
      - CA-ARTICLE-IMPERIAL-CONTROL
    external_depth_routes:
      - archaeology
      - transport_history
      - military_history
      - institutional_history
  unresolved:
    fields:
      - exact_regional_abandonment_dates
    conflicts: []
    schema_gaps: []
  registration_status: PROVISIONAL_REGISTER
  confidence: 0.86

function ingest_and_route(input):
    request =
        create_ingestion_request(input)
    source =
        register_source(input)
    bundle =
        extract(input)
    candidates =
        identify_objects(bundle)
    results = []
    for candidate in candidates:
        identity =
            resolve_identity(candidate)
        object =
            classify(identity)
        object.atlas_position =
            assign_atlas_position(object)
        object.civos =
            map_to_civos(object)
        object.relationships =
            generate_relationships(
                object,
                ATLAS_GRAPH
            )
        object.dependencies =
            detect_dependencies(object)
        object.diagnostics =
            route_diagnostics(object)
        object.provenance =
            attach_provenance(
                input,
                object
            )
        object.confidence =
            calculate_total_confidence(object)
        validation =
            validate_all_schemas(object)
        route_manifest =
            generate_route_manifest(
                object,
                validation
            )
        registration =
            determine_registration_status(
                object.confidence,
                validation
            )
        results.append(
            commit_or_quarantine(
                object,
                route_manifest,
                registration
            )
        )
    return results

function validate_ingestion(result):
    require(valid_object_schema)
    require(valid_scale)
    require(traceable_provenance)
    require(confidence_record)
    require(route_manifest)
    if coordinate_assigned:
        require(valid_coordinate_schema)
    if relationships_assigned:
        require(valid_relationship_schema)
    if external_model:
        require(valid_crosswalk_schema)
    reject(untraceable_generated_claim)
    reject(duplicate_without_resolution)
    reject(permanent_coordinate_without_date)
    reject(causal_edge_without_mechanism)
    reject(auto_registration_below_threshold)
    return VALID

function route_to_articles(object):
    article_matches =
        match(
            object.object_type,
            object.coordinate,
            object.mechanisms,
            object.dependencies,
            object.relationships,
            object.diagnostics
        )
    rank by:
        canonical_relevance
        semantic_relevance
        scale_match
        temporal_match
        evidence_quality
    return ranked_article_ids

ARTICLES DO NOT NEED
TO LIST EVERY POSSIBLE OBJECT.
THE OBJECT FINDS THE ARTICLES
THROUGH SHARED IDS,
SCHEMAS AND RELATIONSHIPS.

function route_user_item(item):
    object =
        ingest_and_route(item)
    return {
        what_it_is:
            object.classification,
        where_it_belongs:
            object.route_manifest,
        what_it_connects_to:
            object.relationships,
        what_it_depends_on:
            object.dependencies,
        what_it_changes:
            object.coordinate_effects,
        where_depth_lives:
            object.external_depth_routes,
        how_certain:
            object.confidence
    }

human_review_triggers:
  - confidence_below_threshold
  - disputed_canonical_identity
  - new_stage_assignment
  - new_habitat_independence_claim
  - major_causal_claim
  - politically_contested_classification
  - schema_extension_required
  - merge_of_high_value_objects
  - contradiction_between_canonical_sources

canonical_rule: >
  Every incoming item must be identified, classified, scaled,
  located, connected, evidenced, validated and routed before
  becoming an active Atlas object. Automation may generate
  provisional classifications, but uncertainty, conflict,
  provenance and review status must remain visible.

THE SCHEMA DEFINES THE POSSIBLE LOCATIONS.
THE INPUT SUPPLIES THE OBJECT.
AI DETECTS THE BEST ADDRESS.
PROVENANCE PRESERVES THE LINEAGE.
CONFIDENCE CONTROLS REGISTRATION.
CONFLICTS ENTER WITHOUT BEING ERASED.
SCHEMA GAPS BECOME DEVELOPMENT SIGNALS.
THE CONTROL TOWER RECEIVES THE ROUTE.
THE ATLAS CAN SCALE
WITHOUT MANUALLY TAGGING
EVERY ITEM IN ADVANCE.