Time Prior Feature - Complete Code Flow Documentation¶

Overview¶

The time prior feature provides time-of-day and day-of-week specific occupancy probabilities. This document provides a comprehensive deep dive into how time priors work, including data sources, calculation, storage, retrieval, and usage in probability calculations.

Implementation Status: Time priors are fully implemented. The system calculates time priors from historical data during the analysis cycle and stores them in the database for efficient retrieval at runtime.

Table of Contents¶

1. Data Sources and Input Flow
2. Time Prior Calculation
3. Storage Mechanism
4. Retrieval at Runtime
5. Usage in Probability Calculation
6. Code Flow Diagrams
7. Implementation Notes

Data Sources and Input Flow¶

1.1 Historical Data Collection¶

Source: Home Assistant Recorder database

Process: sync_states() in db/sync.py imports state history from Home Assistant's recorder

Storage: Raw intervals stored in intervals table with:

• entity_id: The sensor entity ID
• state: The sensor state (e.g., "on", "off")
• start_time: When the state change occurred
• end_time: When the next state change occurred
• duration_seconds: Duration of this interval
• area_name: The area this interval belongs to

Ground Truth: Motion sensors exclusively (see queries.py:get_occupied_intervals())

The system uses motion sensors as the "ground truth" for occupancy because they are the most direct indicator of human presence and rarely generate false positives.

1.2 Occupied Intervals Calculation¶

Location: db/queries.py:get_occupied_intervals()

Method:

1. Queries motion sensor intervals from intervals table filtered by:
2. entry_id: Integration entry ID
3. area_name: Specific area
4. entity_type == InputType.MOTION.value: Only motion sensors
5. state == "on": Only active motion intervals

6. start_time >= lookback_date: Within lookback period

7. Merges overlapping intervals using utils.py:merge_overlapping_intervals()

8. Combines intervals that overlap or are adjacent

9. Prevents double-counting of occupied time

10. Applies motion timeout extension using utils.py:apply_motion_timeout()

11. Extends each PIR-style motion sensor's "on" stretches by the area's motion timeout (default 90 s); sensors that hold "on" (mmWave) are not extended (data/ground_truth.py)
12. Accounts for time after motion stops when area may still be occupied

Output: List of (start_time, end_time) tuples representing occupied periods

Example:

intervals = [
    (datetime(2024, 1, 1, 9, 0), datetime(2024, 1, 1, 9, 15)),
    (datetime(2024, 1, 1, 9, 12), datetime(2024, 1, 1, 9, 30)),  # Overlaps with first
    (datetime(2024, 1, 1, 14, 0), datetime(2024, 1, 1, 14, 5)),
]
# After merging and timeout extension:
merged = [
    (datetime(2024, 1, 1, 9, 0), datetime(2024, 1, 1, 9, 35)),  # Merged + timeout
    (datetime(2024, 1, 1, 14, 0), datetime(2024, 1, 1, 14, 10)),  # Extended by timeout
]

1.3 Caching Layer¶

Table: occupied_intervals_cache (see schema.py:OccupiedIntervalsCache)

Purpose: Precomputed occupied intervals to avoid recalculation

Population: operations.py:save_occupied_intervals_cache()

• Called during analysis cycle
• Stores merged and extended intervals
• Includes calculation_date for cache validity checking

Validity Check: queries.py:is_occupied_intervals_cache_valid()

• Checks if cache is less than 24 hours old (default)
• Returns False if cache is stale or missing

Usage: The cache is used by prior analysis to avoid recalculating occupied intervals from raw sensor data.

Time Prior Calculation¶

2.1 Implementation Overview¶

Time priors are calculated during the analysis cycle by the PriorAnalyzer class. The calculation occurs automatically after global prior calculation, using the same occupied intervals data.

Location: data/analysis.py:PriorAnalyzer.calculate_time_priors()

Grid: 168 time slots (24 hours × 7 days)

• Each slot represents one hour on a specific day of week
• Day of week: 0=Monday, 1=Tuesday, ..., 6=Sunday
• Time slot: 0=00:00-01:00, 1=01:00-02:00, ..., 23=23:00-24:00

2.2 Calculation Process¶

The calculation follows these steps:

1. Get Occupied Intervals: Uses the same occupied intervals calculated for global prior analysis (from motion sensors, merged and extended with motion timeout).

2. Process Each Interval: For each occupied interval:

3. Iterates through all time slots the interval overlaps with
4. Calculates the overlap duration for each slot

5. Tracks which week each overlap belongs to (for data point counting)

6. Calculate Per-Slot Statistics:

7. Occupied Seconds: Sum of all overlap durations for each slot
8. Data Points: Number of unique weeks with data for each slot

9. Total Slot Seconds: data_points × 3600 (seconds per hour)

10. Calculate Prior Values:

11. For each slot: prior_value = occupied_seconds / total_slot_seconds
12. Applies safety bounds: clamps to [TIME_PRIOR_MIN_BOUND, TIME_PRIOR_MAX_BOUND] = [0.03, 0.9]
13. Iterates the denominators (slot_total_seconds), so a slot the analysis period covered but that saw no occupancy is stored at the lower bound. Only slots the period never covered are left unwritten.
14. Counts completed hours only: the period ends at the start of the current local hour. The hour in progress is the live present, not history; counting it let an occupied hour rewrite its own slot at the :57 analysis, and the prior dropped back on the hour.

15. Counts observed time only (#574): a slot's denominator is the time the area was observed: any motion sensor with a recorded state (gaps under OBSERVED_GAP_TOLERANCE_SECONDS = 600 s bridged) or occupied, minus the spans the away-mode entity was on. The global prior uses the same observed time. Unobserved hours count as neither occupied nor empty, and a slot never observed is left unwritten. Floors and All Areas count a moment as observed only while every member room was.

16. Save to Database: Calls save_time_priors() to store all calculated priors with metadata:

17. prior_value: The calculated probability
18. data_points: Number of weeks with data
19. confidence: Calculated as min(1.0, data_points / 4.0) (4 weeks = full confidence)
20. sample_period_start/end: The data period used
21. calculation_method: "interval_analysis"

Note: Only motion sensors are used for time prior calculation (no media/appliance fallback).

Safety Bounds: The system bounds time-specific probabilities between 3% and 90% to prevent extreme values from dominating the calculation.

2.3 Integration Point¶

Called From: data/analysis.py:PriorAnalyzer.calculate_and_update_prior() (line 303-337)

Timing: After global prior calculation completes successfully

Error Handling: If time prior calculation fails, it logs a warning but does not prevent global prior from being saved. The system continues to function using only the global prior until the next analysis cycle.

Storage Mechanism¶

3.1 Database Schema¶

Table: priors (see schema.py:176-247)

Primary Key: Composite of (area_name, day_of_week, time_slot)

• Ensures one prior value per area per time slot
• Allows efficient lookup by area and time slot

Columns:

Indexes:

• idx_priors_entry on entry_id: Fast filtering by integration
• idx_priors_area on area_name: Fast filtering by area
• idx_priors_entry_area on (entry_id, area_name): Composite index for area queries
• idx_priors_day_slot on (day_of_week, time_slot): Fast lookup by time slot
• idx_priors_last_updated on last_updated: For tracking recalculations

Relationships:

• Many-to-one with areas table via area_name foreign key

3.2 Storage Operations¶

Location: db/operations.py:save_time_priors() (line 1000)

Function Signature:

def save_time_priors(
    db: AreaOccupancyDB,
    area_name: str,
    time_priors: dict[tuple[int, int], float],  # (day_of_week, time_slot) -> prior_value
    data_period_start: datetime,
    data_period_end: datetime,
    data_points_per_slot: dict[tuple[int, int], int],  # (day_of_week, time_slot) -> weeks
    calculation_method: str = "interval_analysis",
) -> bool:

Implementation Details:

• Processes all time priors in a single database transaction
• For each slot, checks if a record exists:
• If exists: Updates prior_value, data_points, confidence, last_calculation_date, sample_period_start/end, calculation_method
• If new: Creates a new Priors record with all fields populated
• Calculates confidence as min(1.0, data_points / 4.0) where 4 weeks = full confidence
• Uses bulk operations for efficiency
• Returns True on success, False on error
• Logs debug information about number of slots saved/updated

Retrieval at Runtime¶

4.1 Retrieval Flow¶

The time prior retrieval follows this path:

1. Entry Point: prior.py:Prior.time_prior property (line 115)
2. Called when calculating combined prior

3. Property getter that only reads the cache; it never queries the database

4. Cache Check: Checks if _cached_time_priors dictionary is populated (line 118)

5. If None, returns unlearned_slot_prior without querying the database (the area's global_prior; DEFAULT_TIME_PRIOR (0.5) only before any global prior is learned)
6. Cache stores all time priors as a dictionary: (day_of_week, time_slot) -> prior_value

7. The cache is filled by load_time_priors(), which load_data() and start_prior_analysis() run in the executor, so probability calculations on the event loop never query SQLite

8. Load All Time Priors: prior.py:load_time_priors()

9. Runs in the executor only (load_data on area load, start_prior_analysis after new priors are saved) — time_prior itself never reads the database, since every probability calculation on the event loop calls it
10. Calls db.get_stored_time_priors() to retrieve only the slots actually stored for the area, in a single database query
11. Fills the rest of the weekly grid itself with Prior.unlearned_slot_prior — the area's own global_prior, clamped to [0.03, 0.9] like every time prior. Inside that band it is the identity of combine_priors() and contributes no opinion; a global_prior below 0.03 (or above 0.9) gets the bound instead, which tilts the combined prior slightly toward it. DEFAULT_TIME_PRIOR (0.5) is used only before any global prior exists.
12. Applies safety bounds [TIME_PRIOR_MIN_BOUND, TIME_PRIOR_MAX_BOUND] to stored values during loading, and publishes both maps atomically (the event loop reads while the executor loads)
13. A parallel data_points map is cached alongside; 0 marks an unlearned (filled) slot

14. On a failed database read neither cache is touched — a previously published grid keeps serving and the next load retries

15. Database Method: db/core.py:get_stored_time_priors()

16. Wrapper that adds entry_id parameter

17. Calls query function

18. Query Function: queries.py:get_stored_time_priors()

19. Queries Priors table filtered by:
    • entry_id: Integration entry ID
    • area_name: Area name
20. Returns dictionary mapping (day_of_week, time_slot) to (prior_value, data_points); slots with no stored row are absent

21. Returns None on a database error so the caller can tell a failed read apart from a genuinely empty table

22. Get Current Slot: After cache is loaded, retrieves value for current day/slot (line 121-126)

23. Gets current day_of_week and time_slot
24. Looks up value in cached dictionary

25. Returns unlearned_slot_prior if slot not found (shouldn't happen after load_time_priors())

26. Safety Bounds: Applied during load_time_priors()

27. Clamps all values to [TIME_PRIOR_MIN_BOUND, TIME_PRIOR_MAX_BOUND] = [0.03, 0.9]
28. Prevents extreme values from affecting calculations

4.2 Time Slot Calculation¶

Day of Week: prior.py:day_of_week property (line 156)

• Uses dt_util.utcnow().weekday()
• Returns 0=Monday, 1=Tuesday, ..., 6=Sunday

Time Slot: prior.py:time_slot property (line 161)

• Calculates: (now.hour * 60 + now.minute) // DEFAULT_SLOT_MINUTES
• DEFAULT_SLOT_MINUTES = 60 (line 39)
• Returns 0-23 for hourly slots

Example:

# Monday at 14:30
day_of_week = 0  # Monday
time_slot = (14 * 60 + 30) // 60 = 14  # 14:00-15:00 slot

4.3 Caching Strategy¶

Cache Structure: _cached_time_priors dictionary

• Stores all 168 time priors: dict[tuple[int, int], float]
• Key: (day_of_week, time_slot) tuple
• Value: prior_value (already clamped to [0.03, 0.9], TIME_PRIOR_MIN_BOUND/TIME_PRIOR_MAX_BOUND)
• Loaded once per Prior instance lifecycle

Cache Population:

• Loaded by load_time_priors() in the executor: from load_data() when an area's data is loaded, and from start_prior_analysis() after new time priors are saved
• Loads all 168 slots in a single database query for efficiency
• Applied safety bounds during loading

Cache Invalidation: _invalidate_time_prior_cache() (line 157)

• Called by clear_cache() (area removal and purge)
• Sets _cached_time_priors = None; time_prior returns unlearned_slot_prior until the next load_time_priors()
• set_global_prior() does not invalidate it: the previous snapshot stays in use until the prior analysis publishes the new time priors

TTL: No time-based expiration

• Cache persists until load_time_priors() replaces it or clear_cache() clears it
• Since all 168 slots are cached, no need to check day/slot on each access
• Simply looks up current slot in the cached dictionary

Rationale: Loading all time priors at once is more efficient than individual queries. Time priors don't change frequently (only during analysis cycles), so caching all slots is safe and provides fast access.

Usage in Probability Calculation¶

5.1 Thin slots are shrunk toward the global prior¶

Before a slot is combined, it is mixed with TIME_PRIOR_PSEUDO_WEEKS (2) weeks of the area's global prior, weighted by the weeks of data behind it (forecast.shrink_slot_prior):

slot_used = (weeks × slot + 2 × global_prior) / (weeks + 2)

A slot with one week of history is one afternoon and lands on a bound; on its own it could carry the live prior across the threshold with no sensor active. One week at 0.9 against a 0.228 global becomes 0.452, and the combined prior 0.308 instead of 0.537. After eight weeks the slot's own data carries 80% of the weight. Unlearned slots (0 weeks) already hold the neutral fallback and are used unchanged. The same shrinkage applies to the live prior, prior_for, and the floor / All Areas zone priors; slots_raw in the forecast service still reports the stored value.

5.2 Prior Combination¶

Location: utils.py:combine_priors()

Method: Weighted averaging in logit space

Default Weight: time_weight=0.4 (40% time prior, 60% global prior)

Process:

1. Handle edge cases (time_weight == 0.0 or 1.0)
2. Handle extreme values (0.0 or 1.0) by converting to MIN/MAX_PROBABILITY
3. Convert both priors to logit space: logit(p) = log(p / (1-p))
4. Calculate weighted average: combined_logit = area_weight * area_logit + time_weight * time_logit
5. area_weight = 1.0 - time_weight
6. Convert back to probability: combined_prior = 1 / (1 + exp(-combined_logit))
7. Clamp to valid range [MIN_PROBABILITY, MAX_PROBABILITY]

Why Logit Space?

• Logit space provides better interpolation for probabilities
• Prevents issues with extreme values
• More mathematically principled than linear interpolation

Example:

area_prior = 0.3  # 30% occupancy overall
time_prior = 0.7  # 70% occupancy for this time slot
time_weight = 0.4  # 40% weight to time prior

# Convert to logit space
area_logit = log(0.3 / 0.7) ≈ -0.847
time_logit = log(0.7 / 0.3) ≈ 0.847

# Weighted average
combined_logit = 0.6 * (-0.847) + 0.4 * 0.847 ≈ -0.169

# Convert back
combined_prior = 1 / (1 + exp(0.169)) ≈ 0.458

5.3 Integration Point¶

Location: prior.py:Prior.value property

Flow:

1. Gets global_prior from memory (self.global_prior)
2. Gets time_prior via property (triggers retrieval if needed)
3. Combines using combine_priors() if both exist:
4. If time_prior is None, uses global_prior directly
5. Otherwise combines: combine_priors(global_prior, time_prior)
6. Applies PRIOR_FACTOR (1.05) multiplier
7. Slight bias towards occupancy ("better safe than sorry")
8. Applies min_prior_override if configured
9. Ensures minimum prior doesn't drop too low

Code Flow:

@property
def value(self) -> float:
    if self.global_prior is None:
        result = MIN_PRIOR
    else:
        if self.time_prior is None:
            prior = self.global_prior
        else:
            prior = combine_priors(self.global_prior, self.time_prior)
        # Apply factor and clamping
        adjusted_prior = prior * PRIOR_FACTOR
        result = max(MIN_PRIOR, min(MAX_PRIOR, adjusted_prior))

    # Apply minimum override
    if self.config.min_prior_override > 0.0:
        result = max(result, self.config.min_prior_override)

    return result

Code Flow Diagrams¶

6.1 Time Prior Retrieval Sequence¶

sequenceDiagram
    participant Area as Area Object
    participant Prior as Prior Object
    participant Cache as Cache Dict
    participant DB as Database
    participant Query as Query Function

    Note over Prior,Query: Executor: load_data() / start_prior_analysis() call load_time_priors()
    Prior->>DB: get_stored_time_priors(area_name)
    DB->>Query: get_stored_time_priors(entry_id, area_name)
    Query->>Query: Query stored Priors for area
    Query-->>DB: Stored slots -> (value, data_points), or None on error
    DB-->>Prior: Stored slots (None leaves the cache untouched)
    Prior->>Prior: Fill unstored slots with unlearned_slot_prior; clamp to [0.03, 0.9]
    Prior->>Cache: Publish all 168 slots

    Note over Area,Cache: Event loop: every probability() calculation
    Area->>Prior: probability() calculation
    Prior->>Prior: value property
    Prior->>Prior: time_prior property
    Prior->>Cache: Check if _cached_time_priors is None

    alt Cache Populated
        Cache-->>Prior: Dictionary with all 168 slots
        Prior->>Prior: Get current (day, slot)
        Prior->>Cache: Lookup (day, slot)
        Cache-->>Prior: prior_value for current slot
    else Cache Empty (not loaded yet)
        Prior->>Prior: Use unlearned_slot_prior (no database query)
    end

    Prior->>Prior: combine_priors(global, time)
    Prior-->>Area: Combined prior

6.2 Time Prior Calculation Flow (When Implemented)¶

flowchart TD
    Start([Analysis Cycle]) --> GetIntervals[Get Occupied Intervals]
    GetIntervals --> GroupSlots[Group by Day/Slot]
    GroupSlots --> ForEachSlot{For Each Slot<br/>168 total}

    ForEachSlot --> CalcOccupied[Calculate Occupied Seconds]
    CalcOccupied --> CalcTotal[Calculate Total Slot Seconds]
    CalcTotal --> CalcPercent[Calculate Percentage]
    CalcPercent --> ApplyBounds[Apply Safety Bounds<br/>[0.03, 0.9]]
    ApplyBounds --> StorePrior[Store in Priors Table]

    StorePrior --> MoreSlots{More Slots?}
    MoreSlots -->|Yes| ForEachSlot
    MoreSlots -->|No| Complete([Calculation Complete])

6.3 Complete Prior Calculation Flow¶

flowchart TD
    Start([Prior Calculation]) --> GetIntervals[Get Occupied Intervals<br/>from Motion Sensors]
    GetIntervals --> CalcGlobal[Calculate Global Prior]
    CalcGlobal --> SaveGlobal[Save to GlobalPriors Table]

    SaveGlobal --> CalcTime[Calculate Time Priors]
    CalcTime --> GroupSlots[Group Intervals by<br/>Day of Week & Time Slot]
    GroupSlots --> CalcSlots[Calculate Prior per Slot]
    CalcSlots --> SaveTime[Save to Priors Table]

    SaveTime --> Complete([Prior Calculation Complete])

Implementation Notes¶

7.1 Cache Invalidation Logic¶

Behaviour: The cache holds all 168 slots, so a day/slot rollover needs no invalidation — the lookup simply reads a different key. The cache changes only when the stored values change.

Reload model: reads never repopulate the cache (they run on the event loop). After calculate_and_update_prior() saves new rows, start_prior_analysis reloads the cache via load_time_priors() in the executor. This also removes the historical race where a probability read landing between set_global_prior() and save_time_priors() re-cached the previous run's rows for the next hour.

7.3 Default Value Handling¶

Issue: An unlearned slot has no probability, but the retrieval path has to return one.

Why 0.5 was wrong: A flat DEFAULT_TIME_PRIOR of 0.5 is neutral only in isolation. Fed through combine_priors() against a typical small global prior it becomes higher than every genuinely low-occupancy slot, so "never observed" outranked "observed to be empty" — and the inflation reached the live sensor, not just the forecast. Measured on a 13-area installation, areas whose global prior was 0.01–0.04 were reading a live prior 3× higher purely from unlearned slots.

Current behaviour: unlearned slots fall back to the area's own global_prior (Prior.unlearned_slot_prior), clamped to [0.03, 0.9]. Within that band combine_priors() maps it back to itself, so the slot adds no tilt in either direction; for a global_prior outside it (say 0.01) the fallback is the bound (0.03), a small upward tilt. The parallel data_points map marks these slots with 0 so consumers can render them as no data rather than as a probability — the get_time_priors service exposes it, and the Lovelace card hatches those cells.

7.4 Time Slot Granularity¶

Issue: Fixed 60-minute slots may not match user schedules

Impact: Less granular than ideal for precise time-based learning

Current Behavior: 1-hour slots (24 slots per day × 7 days = 168 slots)

Recommendation: Consider configurable slot duration in future, but 1-hour slots are reasonable default

7.5 Data Period Tracking¶

Status: sample_period_start/end fields are populated by save_time_priors() function

Implementation: The calculation function passes the data period used for analysis, which is stored in the database for tracking and verification purposes.

Related Documentation¶

• Database Schema - Complete database schema documentation
• Prior Learning - User-facing prior learning documentation
• Data Flow - Overall data flow diagrams
• Bayesian Calculation - How priors are used in calculations