{
  "story_id": "76cd079b14112613aefc510ecae7ad4a",
  "desk": "drm3",
  "revision": 2,
  "published_at": "2026-09-03T15:00:04.000Z",
  "content_hash": "e587a30f7a68747eca715b23bf3ec1f05e7f451d42bf405645919e7ce3740e16",
  "hash_basis": "sha256 over `headline\\ndek\\nprose`, plus `\\n` + the canonical citations JSON when any source is placed, plus `\\n#blog` for blogs",
  "basis": {
    "headline": "PySpark Window Functions: Ranking, Totals, and Moving Averages",
    "dek": "A guide to using PySpark window functions for ranking, running totals, and moving averages without collapsing data rows.",
    "prose": "PySpark groupBy() function aggregates data but returns only one row per collection of records, which limits its use when transaction details must be preserved alongside aggregation results. [^1]\n\nPySpark window functions allow users to calculate values across related records without collapsing those records into a single result, enabling the retention of transaction details while gaining aggregation information. [^2] The author experienced a critical data pipeline that ran for over three hours every single day. [^3] The developer concluded that the slowness was caused by the wrong join strategy rather than bad code. [^4]\n\nThe row_number() function assigns a unique sequential number to rows within a partition, while rank() gives tied rows the same rank with gaps, and dense_rank() gives tied rows the same rank without gaps. [^5] Window functions are useful for tasks including ranking rows within groups, calculating running totals, comparing current and previous values, finding each row's share of a group total, and selecting top records from each group. [^6] The developer confirmed that the logic within the pipeline was perfectly clean. [^7]\n\nThe overarching schema for the data pipeline was explicitly correct. [^8]",
    "cited": "[{\"statement\":\"PySpark groupBy() function aggregates data but returns only one row per collection of records, which limits its use when transaction details must be preserved alongside aggregation results.\",\"source\":\"Towards Data Science\",\"instrument\":\"News\",\"claim_key\":null,\"published_at\":\"2026-09-02T11:00:02.000Z\",\"publisher_count\":1,\"sources\":[\"Towards Data Science\"]},{\"statement\":\"PySpark window functions allow users to calculate values across related records without collapsing those records into a single result, enabling the retention of transaction details while gaining aggregation information.\",\"source\":\"Towards Data Science\",\"instrument\":\"News\",\"claim_key\":null,\"published_at\":\"2026-09-02T11:00:02.000Z\",\"publisher_count\":1,\"sources\":[\"Towards Data Science\"]},{\"statement\":\"The author experienced a critical data pipeline that ran for over three hours every single day.\",\"source\":\"dzone.com\",\"instrument\":\"News\",\"claim_key\":null,\"published_at\":\"2026-09-03T15:00:04.000Z\",\"publisher_count\":1,\"sources\":[\"dzone.com\"]},{\"statement\":\"The developer concluded that the slowness was caused by the wrong join strategy rather than bad code.\",\"source\":\"dzone.com\",\"instrument\":\"News\",\"claim_key\":null,\"published_at\":\"2026-09-03T15:00:04.000Z\",\"publisher_count\":1,\"sources\":[\"dzone.com\"]},{\"statement\":\"The row_number() function assigns a unique sequential number to rows within a partition, while rank() gives tied rows the same rank with gaps, and dense_rank() gives tied rows the same rank without gaps.\",\"source\":\"Towards Data Science\",\"instrument\":\"News\",\"claim_key\":null,\"published_at\":\"2026-09-02T11:00:02.000Z\",\"publisher_count\":1,\"sources\":[\"Towards Data Science\"]},{\"statement\":\"Window functions are useful for tasks including ranking rows within groups, calculating running totals, comparing current and previous values, finding each row's share of a group total, and selecting top records from each group.\",\"source\":\"Towards Data Science\",\"instrument\":\"News\",\"claim_key\":null,\"published_at\":\"2026-09-02T11:00:02.000Z\",\"publisher_count\":1,\"sources\":[\"Towards Data Science\"]},{\"statement\":\"The developer confirmed that the logic within the pipeline was perfectly clean.\",\"source\":\"dzone.com\",\"instrument\":\"News\",\"claim_key\":null,\"published_at\":\"2026-09-03T15:00:04.000Z\",\"publisher_count\":1,\"sources\":[\"dzone.com\"]},{\"statement\":\"The overarching schema for the data pipeline was explicitly correct.\",\"source\":\"dzone.com\",\"instrument\":\"News\",\"claim_key\":null,\"published_at\":\"2026-09-03T15:00:04.000Z\",\"publisher_count\":1,\"sources\":[\"dzone.com\"]}]",
    "kind": "news"
  },
  "receipt_verify": "Ed25519 over the dot-joined string `slice_hash.cursor_from.cursor_to.view.view_version.row_count`; public_key and sig are base64url of the raw 32-byte key / 64-byte signature",
  "receipt": null,
  "receipt_note": "this revision predates receipt-keeping (before v0.37.0); the filed row lives in the record",
  "generation_chain": {
    "wire": {
      "stream": "fountain_news",
      "story_id": "596788f3c35068f57b09325a4faffa1a",
      "thread_id": "2097a33906c283731b9e63baf43fe494",
      "thread_label": "PySpark",
      "novelty": "UPDATE",
      "content_hash": "9ed1cb21b8de74ed9a3bb782e08891a4720f87289891a3b559efc3506b4a54da",
      "last_published_at": "2026-09-03T15:00:04.000Z",
      "read_receipt": {
        "slice_hash": "28702f8a18e817ad8518c639092f19d5070ea2f9a8aae7ca3b76094e6bf4faa9",
        "cursor_from": "eyJ0cyI6IjIwMjYtMDktMDNUMTQ6NDE6MzEuMDAwMDAwWiIsImlkIjoiYzY1ODgxMmEzMDljNTIzOTBhNTA2MDk2MzMxNDEyMmIiLCJ2IjoiMSJ9",
        "cursor_to": "eyJ0cyI6IjIwMjYtMDktMDNUMTU6MjA6MDAuMDAwMDAwWiIsImlkIjoiZTE0ODFkZDgwOTYxOTkyM2VkNWNmODU3ZDJkZmRlNmEiLCJ2IjoiMSJ9",
        "view": "v_fountain_news",
        "view_version": "1",
        "row_count": 100,
        "window_days": 3,
        "bytes_scanned": 12166856,
        "credits": 8,
        "price_per_100_rows": 8,
        "sig": "jB1UAQDGwfspWj4WCdYK5x4yoX7nVEoMyKNMANFAk699HwKo9KKNbDwlmmZBD0z6dPqJCLKqikYPVYyDeO9-CQ",
        "public_key": "bMUigy8O0jOnBxQ4Sc-5lwhIZ8LQVAhxMbR7qESVuUE",
        "signer_path": "lakehouse/data-extract/v1",
        "alg": "Ed25519",
        "signed": true
      },
      "compose": 3,
      "recomposed_at": "2026-09-05T20:45:45.313Z"
    },
    "written_at": "2026-09-04T06:24:06.618Z"
  },
  "cited_facts": [
    {
      "n": 1,
      "statement": "PySpark groupBy() function aggregates data but returns only one row per collection of records, which limits its use when transaction details must be preserved alongside aggregation results.",
      "source": "Towards Data Science",
      "instrument": "News",
      "claim_key": null,
      "url": null,
      "occurred_at": null,
      "published_at": null
    },
    {
      "n": 2,
      "statement": "PySpark window functions allow users to calculate values across related records without collapsing those records into a single result, enabling the retention of transaction details while gaining aggregation information.",
      "source": "Towards Data Science",
      "instrument": "News",
      "claim_key": null,
      "url": null,
      "occurred_at": null,
      "published_at": null
    },
    {
      "n": 3,
      "statement": "The author experienced a critical data pipeline that ran for over three hours every single day.",
      "source": "dzone.com",
      "instrument": "News",
      "claim_key": null,
      "url": null,
      "occurred_at": null,
      "published_at": null
    },
    {
      "n": 4,
      "statement": "The developer concluded that the slowness was caused by the wrong join strategy rather than bad code.",
      "source": "dzone.com",
      "instrument": "News",
      "claim_key": null,
      "url": null,
      "occurred_at": null,
      "published_at": null
    },
    {
      "n": 5,
      "statement": "The row_number() function assigns a unique sequential number to rows within a partition, while rank() gives tied rows the same rank with gaps, and dense_rank() gives tied rows the same rank without gaps.",
      "source": "Towards Data Science",
      "instrument": "News",
      "claim_key": null,
      "url": null,
      "occurred_at": null,
      "published_at": null
    },
    {
      "n": 6,
      "statement": "Window functions are useful for tasks including ranking rows within groups, calculating running totals, comparing current and previous values, finding each row's share of a group total, and selecting top records from each group.",
      "source": "Towards Data Science",
      "instrument": "News",
      "claim_key": null,
      "url": null,
      "occurred_at": null,
      "published_at": null
    },
    {
      "n": 7,
      "statement": "The developer confirmed that the logic within the pipeline was perfectly clean.",
      "source": "dzone.com",
      "instrument": "News",
      "claim_key": null,
      "url": null,
      "occurred_at": null,
      "published_at": null
    },
    {
      "n": 8,
      "statement": "The overarching schema for the data pipeline was explicitly correct.",
      "source": "dzone.com",
      "instrument": "News",
      "claim_key": null,
      "url": null,
      "occurred_at": null,
      "published_at": null
    }
  ],
  "note": "A signature proves who filed this and that it has not changed since. It never makes a claim true. The dates on cited_facts (occurred_at, published_at) are the forge's, joined at read time; they are not part of the signed basis above."
}