{
  "slug": "PostgreSQL-Concurrency--Locking--and-Isolation-Levels-9fde2d582668",
  "title": "PostgreSQL Concurrency, Locking, and Isolation Levels",
  "subtitle": "1. Optimistic vs. Pessimistic Locking:\n — Pessimistic: Assumes worst-case scenario\n — Optimistic: Assumes best-case scenario",
  "excerpt": "1. Optimistic vs. Pessimistic Locking:\n — Pessimistic: Assumes worst-case scenario\n — Optimistic: Assumes best-case scenario",
  "date": "2024-08-28",
  "tags": [
    "Postgres"
  ],
  "readingTime": "6 min",
  "url": "https://medium.com/@mobinshaterian/postgresql-concurrency-locking-and-isolation-levels-9fde2d582668",
  "hero": "https://cdn-images-1.medium.com/max/800/1*Prhs8vu83OET5xc7O8DgWA.png",
  "content": [
    {
      "type": "paragraph",
      "html": "1. Optimistic vs. Pessimistic Locking:<br> — Pessimistic: Assumes worst-case scenario<br> — Optimistic: Assumes best-case scenario"
    },
    {
      "type": "paragraph",
      "html": "2. Transaction Isolation Levels in PostgreSQL:<br>&nbsp;a. Read Committed (default):<br> — Reads: Can see updates from other committed transactions<br> — Writes: Waits for conflicting transactions to commit or abort"
    },
    {
      "type": "paragraph",
      "html": "b. Repeatable Read:<br> — Reads: Cannot see writes from transactions that started later<br> — Writes: Throws a serializable error if the conflicting transaction commits"
    },
    {
      "type": "paragraph",
      "html": "c. Serializable:<br> — The highest level of isolation<br> — Uses Serializable Snapshot Isolation (SSI)<br> — Ensures strictest data consistency"
    },
    {
      "type": "paragraph",
      "html": "3. ACID Compliance:<br> — PostgreSQL: Fully ACID-compliant<br> — Redis: Not fully ACID-compliant (lacks consistency and durability)<br> — MongoDB: ACID-compliant with multi-document transactions<br> — Cassandra: Not ACID-compliant in the traditional sense"
    },
    {
      "type": "paragraph",
      "html": "4. Concurrency Issues:<br> — Double booking problem<br> — Time-of-check to time-of-use (TOCTOU) bugs"
    },
    {
      "type": "paragraph",
      "html": "5. Solid Queue:<br> — Database-based queuing backend for Active Job<br> — Uses row-level locking (FOR UPDATE) to manage concurrent access"
    },
    {
      "type": "paragraph",
      "html": "6. Locking Options:<br> — FOR UPDATE: Locks rows, causing other transactions to wait<br> — SKIP LOCKED: Allows transactions to skip locked rows and move to the next available one<br> — NOWAIT: Throws an exception immediately if a row is locked"
    },
    {
      "type": "paragraph",
      "html": "7. Common Concurrency Problems:<br> — Lost updates<br> — Write skew<br> — Race conditions"
    },
    {
      "type": "paragraph",
      "html": "This report provides an overview of PostgreSQL’s approach to handling concurrent transactions, its isolation levels, and a comparison with other database systems regarding ACID compliance. It also touches on related concepts like queue management and locking strategies to prevent concurrency issues."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "postgres optimistic vs pessimistic locking"
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/I8IlO0hCSgY?feature=oembed"
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/pomxJOFVcQs?feature=oembed"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "optimistic VS pessimistic"
    },
    {
      "type": "paragraph",
      "html": "Pessimistic: The worst case is going on to happen."
    },
    {
      "type": "paragraph",
      "html": "Optimistic: The best case is going on to happen."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Race Condition"
    },
    {
      "type": "paragraph",
      "html": "Access to one row in the database"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*Prhs8vu83OET5xc7O8DgWA.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 531,
      "height": 290
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Transaction"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*8EcFEODqiRTTh9eKyys1kg.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1364,
      "height": 725
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Atomicity"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*51CIwzsg8JYQ884FOo1OVg.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1364,
      "height": 725
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Isolation"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*TJ8FesJuVTf5bTp0J4SLsg.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1270,
      "height": 474
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Isolation level"
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/G8wDjV0N9tk?feature=oembed"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*hhWm5sKuAqL13qTgO8xjGg.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 472,
      "height": 333
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*gWkUa27m2rSnfb2lz6cy1A.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1395,
      "height": 724
    },
    {
      "type": "heading",
      "level": 3,
      "text": "It could happen in PostgreSQL."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*C-_01uSVdki_GhLCeQuB1A.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 896,
      "height": 380
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*a0c1WtNT5S53MN4F7SmLaA.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 896,
      "height": 380
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*9tHnSzJHqOdFS81MkLZqwA.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1395,
      "height": 724
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*t0ENCjfeJ6gLD_pqsMDT7w.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 896,
      "height": 380
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*wcdKCgcOA8fhsZfVjBsR1Q.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1395,
      "height": 724
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*fXVZJ-MiSnlWn-R56w0Awg.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 909,
      "height": 422
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*uawc3EeFPbzE2qri1aOk3Q.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1347,
      "height": 506
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*78Xyfn1LNojP5gkqr-4qeA.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1347,
      "height": 506
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*3oYOkes8t2eXozoGtb-6-w.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 911,
      "height": 489
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "When an application process accesses data, the isolation level determines the degree to which that data is locked or isolated from other concurrent processes.",
        "Postgres doesn’t have a read uncommitted isolation level, and it also doesn’t allow dirty reads or writes due to its MVCC architecture. (will cover this in the next article)"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Read committed (default)"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Reads:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "When multiple transactions are running concurrently, a transaction <strong>can see the updates</strong> made by other transactions already been committed.",
        "These updates can be seen even if the other transactions started and committed while the current transaction was still ongoing."
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Writes:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "If a transaction attempts to update a row that has already been updated by another concurrent transaction (which has not yet been committed), that transaction will pause and wait until the first transaction either commits or aborts.",
        "After the first concurrent transaction commits or aborts, the paused transaction can continue with its write."
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Repeatable read"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Reads:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "When multiple transactions are running concurrently, a transaction <strong>cannot see the writes</strong> made by other transactions that started after the current transaction.",
        "This applies even if those transactions have already been committed while the current transaction is in progress."
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Writes:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "If a transaction attempts to update a row that has already been updated by another concurrent transaction (which has not yet been committed), it will wait until the first transaction either commits or aborts.",
        "However, if the concurrent transaction commits its changes, the paused transaction will throw a serializable error.",
        "Essentially, this isolation level ensures that no two transactions can update the same row concurrently."
      ]
    },
    {
      "type": "paragraph",
      "html": "If we try out the lost update use case mentioned above, the 2nd transaction will throw the following error"
    },
    {
      "type": "quote",
      "html": "<em>ERROR: could not serialize access due to concurrent update</em>"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Serializable"
    },
    {
      "type": "paragraph",
      "html": "This is the highest level of transaction isolation and ensures the strictest level of data consistency. It mimics executing transactions in a serial manner, meaning one after the other instead of concurrently."
    },
    {
      "type": "paragraph",
      "html": "Transactions will only be committed if their writings do not affect the result if executed in a different order."
    },
    {
      "type": "paragraph",
      "html": "Postgres uses Serializable Snapshot Isolation (SSI) to achieve this isolation (will cover this in detail in next article)."
    },
    {
      "type": "paragraph",
      "html": "SSI achieves this isolation level in 2 ways:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "If a transaction has performed some read queries before a write operation, it performs that read query again while committing and makes sure the result set has not changed. If it has changed, the transaction is aborted.",
        "A writing transaction informs other reading transactions that the latter’s earlier executed search query might have been messed up. In this case, the transaction that commits first would win, and the other one gets aborted."
      ]
    },
    {
      "type": "paragraph",
      "html": "If we try out the write skew use case mentioned above, the 2nd transaction will throw the following error."
    },
    {
      "type": "quote",
      "html": "<em>ERROR: could not serialize access due to read/write dependencies among transactions. DETAIL: Reason code: Canceled on identification as a pivot, during commit attempt.</em>"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "ACID"
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/GAe5oB742dw?feature=oembed"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*mvIwptz7qIKwvUQQASg3Wg.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 1307,
      "height": 424
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/pomxJOFVcQs?feature=oembed"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*RNK48ZdtxSoC97EsKpW61w.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 323,
      "height": 305
    },
    {
      "type": "paragraph",
      "html": "Redis is not completely ACID-compliant because it does not satisfy consistency nor durability. Redis ensures atomicity because you cannot go to the next transaction till your previous transactions are fully completed. Redis does not guarantee consistency because Redis is at risk of losing writes"
    },
    {
      "type": "paragraph",
      "html": "In fact, <strong>MongoDB is an ACID-compliant database</strong>. As of MongoDB 4.0, there is even support for multi-document ACID transactions when required. Version 4.2 even brought distributed multi-document ACID transactions for even more flexibility."
    },
    {
      "type": "paragraph",
      "html": "As a non-relational database, Cassandra does not support joins or foreign keys, and consequently <strong>does not offer consistency in the ACID sense</strong>. For example, when moving money from account A to B the total in the accounts does not change."
    },
    {
      "type": "paragraph",
      "html": "<a href=\"https://docs.datastax.com/en/cassandra-oss/2.2/cassandra/dml/dmlTransactionsDiffer.html\" target=\"_blank\" rel=\"noreferrer noopener\">https://docs.datastax.com/en/cassandra-oss/2.2/cassandra/dml/dmlTransactionsDiffer.html</a>"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Double booking problem"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*lc3bfKaQUdwWXjQ1SvkW6Q.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 531,
      "height": 290
    },
    {
      "type": "heading",
      "level": 2,
      "text": "time-of-check to time-of-use"
    },
    {
      "type": "paragraph",
      "html": "In <a href=\"https://en.wikipedia.org/wiki/Software_development\" target=\"_blank\" rel=\"noreferrer noopener\">software development</a>, <strong>time-of-check to time-of-use</strong> (<strong>TOCTOU</strong>, <strong>TOCTTOU</strong> or <strong>TOC/TOU</strong>) is a class of <a href=\"https://en.wikipedia.org/wiki/Software_bug\" target=\"_blank\" rel=\"noreferrer noopener\">software bugs</a> caused by a <a href=\"https://en.wikipedia.org/wiki/Race_condition\" target=\"_blank\" rel=\"noreferrer noopener\">race condition</a> involving the <em>checking</em> of the state of a part of a system (such as a security credential) and the <em>use</em> of the results of that check."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "What is solid queue?"
    },
    {
      "type": "paragraph",
      "html": "Solid Queue is database based queuing backend for Active Job. In contrast Sidekiq and Resque are Redis based queuing backends."
    },
    {
      "type": "paragraph",
      "html": "The first worker will take the lock on the record using FOR UPDATE. When other workers come to that record and they see that there is a lock FOR UPDATE, they will wait for the lock to be lifted. Yes, these workers will wait until the lock is released."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "SKIP LOCKED skips locked rows"
    },
    {
      "type": "code",
      "lang": "sql",
      "code": "START TRANSACTION;\nSELECT *\nFROM jobs_table FOR UPDATE SKIP LOCKED\nLIMIT 1;\n-- Process the jobCOMMIT;"
    },
    {
      "type": "paragraph",
      "html": "Imagine the same scenrio here. A job comes in. Multiple workers compete to claim the job. The database ensures that only one worker gets the lock. However in this case the other workers will move on to the next record. They will not wait. That’s what SKIP LOCKED does."
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/m6-63kpttQk"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*xHlvJEmD7vg--4kat-WvlA.png",
      "alt": "PostgreSQL Concurrency, Locking, and Isolation Levels",
      "caption": "",
      "width": 946,
      "height": 465
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/nuBi2XbHH18?feature=oembed"
    },
    {
      "type": "paragraph",
      "html": "When a transaction attempts to read or lock a row, it may find that another transaction has already locked that row. By default, the new transaction will wait until the previous transaction completes. However, there are two ways to override this behaviour:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "NOWAIT: This option allows a transaction to skip the wait and immediately throw an exception when it finds the row is already locked by another transaction.",
        "SKIP LOCKED: This option allows a transaction to skip over any rows locked by other concurrent transactions. As a result, the result set obtained may not contain all the rows requested, but it can complete more quickly without waiting for other transactions to release the locks."
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Common problems in concurrent transactions"
    },
    {
      "type": "paragraph",
      "html": "Visit us at <a href=\"https://www.datadriveninvestor.com/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>DataDrivenInvestor.com</em></a>"
    },
    {
      "type": "paragraph",
      "html": "Subscribe to DDIntel <a href=\"https://www.ddintel.com/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>here</em></a>."
    },
    {
      "type": "paragraph",
      "html": "Join our creator ecosystem <a href=\"https://join.datadriveninvestor.com/\" target=\"_blank\" rel=\"noreferrer noopener\"><em>here</em></a>."
    },
    {
      "type": "paragraph",
      "html": "DDI Official Telegram Channel: <a href=\"https://t.me/+tafUp6ecEys4YjQ1\" target=\"_blank\" rel=\"noreferrer noopener\">https://t.me/+tafUp6ecEys4YjQ1</a>"
    },
    {
      "type": "paragraph",
      "html": "Follow us on <a href=\"https://www.linkedin.com/company/data-driven-investor\" target=\"_blank\" rel=\"noreferrer noopener\"><em>LinkedIn</em></a>, <a href=\"https://twitter.com/@DDInvestorHQ\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Twitter</em></a>, <a href=\"https://www.youtube.com/c/datadriveninvestor\" target=\"_blank\" rel=\"noreferrer noopener\"><em>YouTube</em></a>, and <a href=\"https://www.facebook.com/datadriveninvestor\" target=\"_blank\" rel=\"noreferrer noopener\"><em>Facebook</em></a>."
    }
  ]
}