PostgreSQL Weekly News - June 27, 2021

Posted on 2021-06-28 by PWN
PWN

PostgreSQL Weekly News - June 27, 2021

PostgreSQL 14 Beta 2 released. Test!

The 12th PostgreSQL Conference Cuba (PostgresqlCUBA, @PgCuba) will take place on November 18-19, 2021, at the Hotel Habana Libre. This event is part of the TECNOGET conference, and we will host one track dedicated to PostgreSQL related talks. For more details, reach out to cu AT postgresql DOT org.

PostgreSQL Product News

JDBC 42.2.22 released

pgpoolAdmin 4.2.0, the administration tool for Pgpool-II, released.

pgCenter 0.9.0, a command-line admin tool for observing and troubleshooting PostgreSQL, released

PostgreSQL Jobs for June

https://archives.postgresql.org/pgsql-jobs/2021-06/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Tom Lane pushed:

  • Work around portability issue with newer versions of mktime(). Recent glibc versions have made mktime() fail if tm_isdst is inconsistent with the prevailing timezone; in particular it fails for tm_isdst = 1 when the zone is UTC. (This seems wildly inconsistent with the POSIX-mandated treatment of "incorrect" values for the other fields of struct tm, so if you ask me it's a bug, but I bet they'll say it's intentional.) This has been observed to cause cosmetic problems when pg_restore'ing an archive created in a different timezone. To fix, do mktime() using the field values from the archive, and if that fails try again with tm_isdst = -1. This will give a result that's off by the UTC-offset difference from the original zone, but that was true before, too. It's not terribly critical since we don't do anything with the result except possibly print it. (Someday we should flush this entire bit of logic and record a standard-format timestamp in the archive instead. That's not okay for a back-patched bug fix, though.) Also, guard our only other use of mktime() by having initdb's build_time_t() set tm_isdst = -1 not 0. This case could only have an issue in zones that are DST year-round; but I think some do exist, or could in future. Per report from Wells Oliver. Back-patch to all supported versions, since any of them might need to run with a newer glibc. Discussion: https://postgr.es/m/CAOC+FBWDhDHO7G-i1_n_hjRzCnUeFO+H-Czi1y10mFhRWpBrew@mail.gmail.com https://git.postgresql.org/pg/commitdiff/f807e3410fdfc29ced6590c7c2afa76637e001ad

  • Remove orphaned expected-result file. This should have been removed in 43e084197, which removed the corresponding spec file. Noted while fooling about with the isolationtester. https://git.postgresql.org/pg/commitdiff/ffbe9dec13599fa786ea6567df1c6a3f3ee3c673

  • Update variant expected-result file. This should have been updated in d2d8a229b, but it was overlooked. According to 31a877f18 which added it, this file is meant to show the results you get under default_transaction_isolation = serializable. We've largely lost track of that goal in other isolation tests, but as long as we've got this one, it should be right. Noted while fooling about with the isolationtester. https://git.postgresql.org/pg/commitdiff/0a1e80c5c4f094087257fc4284a87e0bc7bca591

  • Remove another orphan expected-result file. aborted-keyrevoke_2.out was apparently needed when it was added (in commit 0ac5ad513) to handle the case of serializable transaction mode. However, the output in serializable mode actually matches the regular aborted-keyrevoke.out file, and AFAICT has done so for a long time. There's no need to keep dragging this variant along. https://git.postgresql.org/pg/commitdiff/f6352a0d4e437ac8bc266f77df22d064592056c9

  • Update another variant expected-result file. This should have been updated in 533e9c6b0, but it was overlooked. Given the lack of complaints, I won't bother back-patching. https://git.postgresql.org/pg/commitdiff/d3c878499c9d639ff06e0664d06b8c731e30c2fc

  • Improve SQLSTATE reporting in some replication-related code. I started out with the goal of reporting ERRCODE_CONNECTION_FAILURE when walrcv_connect() fails, but as I looked around I realized that whoever wrote this code was of the opinion that errcodes are purely optional. That's not my understanding of our project policy. Hence, make sure that an errcode is provided in each ereport that (a) is ERROR or higher level and (b) isn't arguably an internal logic error. Also fix some very dubious existing errcode assignments. While this is not per policy, it's also largely cosmetic, since few of these cases could get reported to applications. So I don't feel a need to back-patch. Discussion: https://postgr.es/m/2189704.1623512522@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/6b787d9e32005867ee3660d1ea20f447810a403d

  • Fix plancache refcount leak after error in ExecuteQuery. When stuffing a plan from the plancache into a Portal, one is not supposed to risk throwing an error between GetCachedPlan and PortalDefineQuery; if that happens, the plan refcount incremented by GetCachedPlan will be leaked. I managed to break this rule while refactoring code in 9dbf2b7d7. There is no visible consequence other than some memory leakage, and since nobody is very likely to trigger the relevant error conditions many times in a row, it's not surprising we haven't noticed. Nonetheless, it's a bug, so rearrange the order of operations to remove the hazard. Noted on the way to looking for a better fix for bug #17053. This mistake is pretty old, so back-patch to all supported branches. https://git.postgresql.org/pg/commitdiff/131ea3e908d3c97a2fe1ab25cce5046dd5cb905f

  • Centralize the logic for protective copying of utility statements. In the "simple Query" code path, it's fine for parse analysis or execution of a utility statement to scribble on the statement's node tree, since that'll just be thrown away afterwards. However it's not fine if the node tree is in the plan cache, as then it'd be corrupted for subsequent executions. Up to now we've dealt with that by having individual utility-statement functions apply copyObject() if they were going to modify the tree. But that's prone to errors of omission. Bug #17053 from Charles Samborski shows that CREATE/ALTER DOMAIN didn't get this memo, and can crash if executed repeatedly from plan cache. In the back branches, we'll just apply a narrow band-aid for that, but in HEAD it seems prudent to have a more principled fix that will close off the possibility of other similar bugs in future. Hence, let's hoist the responsibility for doing copyObject up into ProcessUtility from its children, thus ensuring that it happens for all utility statement types. Also, modify ProcessUtility's API so that its callers can tell it whether a copy step is necessary. It turns out that in all cases, the immediate caller knows whether the node tree is transient, so this doesn't involve a huge amount of code thrashing. In this way, while we lose a little bit in the execute-from-cache code path due to sometimes copying node trees that wouldn't be mutated anyway, we gain something in the simple-Query code path by not copying throwaway node trees. Statements that are complex enough to be expensive to copy are almost certainly ones that would have to be copied anyway, so the loss in the cache code path shouldn't be much. (Note that this whole problem applies only to utility statements. Optimizable statements don't have the issue because we long ago made the executor treat Plan trees as read-only. Perhaps someday we will make utility statement execution act likewise, but I'm not holding my breath.) Discussion: https://postgr.es/m/931771.1623893989@sss.pgh.pa.us Discussion: https://postgr.es/m/17053-3ca3f501bbc212b4@postgresql.org https://git.postgresql.org/pg/commitdiff/7c337b6b527b7052e6a751f966d5734c56f668b5

  • Improve version reporting in pgbench. Commit 547f04e73 caused pgbench to start printing its version number, which seems like a fine idea, but it needs a bit more work: * Print the server version number too, when different. * Print the PG_VERSION string, not some reconstructed approximation. This patch copies psql's well-tested code for the same purpose. Discussion: https://postgr.es/m/1226654.1624036821@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/84bee9610965331d5110971d8de390a5bbe2effc

  • Fix misbehavior of DROP OWNED BY with duplicate polroles entries. Ordinarily, a pg_policy.polroles array wouldn't list the same role more than once; but CREATE POLICY does not prevent that. If we perform DROP OWNED BY on a role that is listed more than once, RemoveRoleFromObjectPolicy either suffered an assertion failure or encountered a tuple-updated-by-self error. Rewrite it to cope correctly with duplicate entries, and add a CommandCounterIncrement call to prevent the other problem. Per discussion, there's other cleanup that ought to happen here, but this seems like the minimum essential fix. Per bug #17062 from Alexander Lakhin. It's been broken all along, so back-patch to all supported branches. Discussion: https://postgr.es/m/17062-11f471ae3199ca23@postgresql.org https://git.postgresql.org/pg/commitdiff/d21fca084356946664bfce19d66d2df2bb873cbd

  • Provide feature-test macros for libpq features added in v14. We had a request to provide a way to test at compile time for the availability of the new pipeline features. More generally, it seems like a good idea to provide a way to test via #ifdef for all new libpq API features. People have been using the version from pg_config.h for that; but that's more likely to represent the server version than the libpq version, in the increasingly-common scenario where they're different. It's safer if libpq-fe.h itself is the source of truth about what features it offers. Hence, establish a policy that starting in v14 we'll add a suitable feature-is-present macro to libpq-fe.h when we add new API there. (There doesn't seem to be much point in applying this policy retroactively, but it's not too late for v14.) Tom Lane and Alvaro Herrera, per suggestion from Boris Kolpackov. Discussion: https://postgr.es/m/boris.20210617102439@codesynthesis.com https://git.postgresql.org/pg/commitdiff/6991e774e0304f5ef488cf1ae4fa79578b6ae3d5

  • Stabilize test case added by commit f61db909d. Buildfarm members ayu and tern have sometimes shown a different plan than expected for this query. I'd been unable to reproduce that before today, but I finally realized what is happening. If there is a concurrent open transaction (probably an autovacuum run in the buildfarm, but this can also be arranged manually), then the index entries for the rows removed by the DELETE a few lines up are not killed promptly, causing a change in the planner's estimate of the extremal value of ft2.c1, which moves the rowcount estimate for "c1 > 1100" by enough to change the join plan from nestloop to hash. To fix, change the query condition to "c1 > 1000", causing the hash plan to be preferred whether or not a concurrent open transaction exists. Since this UPDATE is tailored to be a no-op, nothing else changes. Report: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=ayu&dt=2021-06-09%2022%3A45%3A48 Report: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=ayu&dt=2021-06-13%2022%3A38%3A18 Report: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2021-06-20%2004%3A55%3A36 https://git.postgresql.org/pg/commitdiff/5843659d091bfb6f2c60e010ea1fd00e55ee6ada

  • Restore the portal-level snapshot for simple expressions, too. Commits 84f5c2908 et al missed the need to cover plpgsql's "simple expression" code path. If the first thing we execute after a COMMIT/ROLLBACK is one of those, rather than a full-fledged SPI command, we must explicitly do EnsurePortalSnapshotExists() to make sure we have an outer snapshot. Note that it wouldn't be good enough to just push a snapshot for the duration of the expression execution: what comes back might be toasted, so we'd better have a snapshot protecting it. The test case demonstrating this fact cheats a bit by marking a SQL function immutable even though it fetches from a table. That's nothing that users haven't been seen to do, though. Per report from Jim Nasby. Back-patch to v11, like the previous fix. Discussion: https://postgr.es/m/378885e4-f85f-fc28-6c91-c4d1c080bf26@amazon.com https://git.postgresql.org/pg/commitdiff/d102aafb6259a6a412803d4b1d8c4f00aa17f67e

  • Use annotations to reduce instability of isolation-test results. We've long contended with isolation test results that aren't entirely stable. Some test scripts insert long delays to try to force stable results, which is not terribly desirable; but other erratic failure modes remain, causing unrepeatable buildfarm failures. I've spent a fair amount of time trying to solve this by improving the server-side support code, without much success: that way is fundamentally unable to cope with diffs that stem from chance ordering of arrival of messages from different server processes. We can improve matters on the client side, however, by annotating the test scripts themselves to show the desired reporting order of events that might occur in different orders. This patch adds three types of annotations to deal with (a) test steps that might or might not complete their waits before the isolationtester can see them waiting; (b) test steps in different sessions that can legitimately complete in either order; and (c) NOTIFY messages that might arrive before or after the completion of a step in another session. We might need more annotation types later, but this seems to be enough to deal with the instabilities we've seen in the buildfarm. It also lets us get rid of all the long delays that were previously used, cutting more than a minute off the runtime of the isolation tests. Back-patch to all supported branches, because the buildfarm instabilities affect all the branches, and because it seems desirable to keep isolationtester's capabilities the same across all branches to simplify possible future back-patching of tests. Discussion: https://postgr.es/m/327948.1623725828@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/741d7f1047fe52da7ced6fa9cea661ce9320c8d4

  • Improve display of query results in isolation tests. Previously, isolationtester displayed SQL query results using some ad-hoc code that clearly hadn't had much effort expended on it. Field values longer than 14 characters weren't separated from the next field, and usually caused misalignment of the columns too. Also there was no visual separation of a query's result from subsequent isolationtester output. This made test result files confusing and hard to read. To improve matters, let's use libpq's PQprint() function. Although that's long since unused by psql, it's still plenty good enough for the purpose here. Like 741d7f104, back-patch to all supported branches, so that this isn't a stumbling block for back-patching isolation test changes. Discussion: https://postgr.es/m/582362.1623798221@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/4a054069a36032a59afceb07f3b837f09ab1a2e9

  • Don't assume GSSAPI result strings are null-terminated. Our uses of gss_display_status() and gss_display_name() assumed that the gss_buffer_desc strings returned by those functions are null-terminated. It appears that they generally are, given the lack of field complaints up to now. However, the available documentation does not promise this, and some man pages for gss_display_status() show examples that rely on the gss_buffer_desc.length field instead of expecting null termination. Also, we now have a report that on some implementations, clang's address sanitizer is of the opinion that the byte after the specified length is undefined. Hence, change the code to rely on the length field instead. This might well be cosmetic rather than fixing any real bug, but it's hard to be sure, so back-patch to all supported branches. While here, also back-patch the v12 changes that made pg_GSS_error deal honestly with multiple messages available from gss_display_status. Per report from Sudheer H R. Discussion: https://postgr.es/m/5372B6D4-8276-42C0-B8FB-BD0918826FC3@tekenlight.com https://git.postgresql.org/pg/commitdiff/126cdaf47af275f76b2f2ddb023bfdc6f018ae30

  • Doc: fix confusion about LEAKPROOF in syntax summaries. The syntax summaries for CREATE FUNCTION and allied commands made it look like LEAKPROOF is an alternative to IMMUTABLE/STABLE/VOLATILE, when of course it is an orthogonal option. Improve that. Per gripe from aazamrafeeque0. Thanks to David Johnston for suggestions. Discussion: https://postgr.es/m/162444349581.694.5818572718530259025@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/2031e1668e5577e64cfed29da69a34903d5a5227

  • Allow non-quoted identifiers as isolation test session/step names. For no obvious reason, isolationtester has always insisted that session and step names be written with double quotes. This is fairly tedious and does little for test readability, especially since the names that people actually choose almost always look like normal identifiers. Hence, let's tweak the lexer to allow SQL-like identifiers not only double-quoted strings. (They're SQL-like, not exactly SQL, because I didn't add any case-folding logic. Also there's no provision for U&"..." names, not that anyone's likely to care.) There is one incompatibility introduced by this change: if you write "foo""bar" with no space, that used to be taken as two identifiers, but now it's just one identifier with an embedded quote mark. I converted all the src/test/isolation/ specfiles to remove unnecessary double quotes, but stopped there because my eyes were glazing over already. Like 741d7f104, back-patch to all supported branches, so that this isn't a stumbling block for back-patching isolation test changes. Discussion: https://postgr.es/m/759113.1623861959@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/a443c1b2d6a646cf90a8afc193c07ed12a2bf045

  • Further stabilize postgres_fdw test. The queries involving ft1_nopw don't stably return the same row anymore. I surmise that an autovacuum hitting "S 1"."T 1" right after the updates introduced by f61db909d/5843659d0 freed some space, changing where subsequent insertions get stored. It's only by good luck that these results were stable before, though, since a LIMIT without ORDER BY isn't well defined, and it's not like we've ever treated that table as append-only in this test script. Since we only really care whether these commands succeed or not, just replace "SELECT *" with "SELECT 1". Report: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2021-06-23%2019%3A52%3A08 https://git.postgresql.org/pg/commitdiff/802177090992511c610804da54a4603d4f50c594

  • Doc: remove commit f560209c6 from v14 release notes. Now that this has been back-patched, it's no longer a new feature for v14. https://git.postgresql.org/pg/commitdiff/8a80562d732c0da1ddcc9fb88dfb976f4b846577

  • Remove unnecessary failure cases in RemoveRoleFromObjectPolicy(). It's not really necessary for this function to open or lock the relation associated with the pg_policy entry it's modifying. The error checks it's making on the rel are if anything counterproductive (e.g., if we don't want to allow installation of policies on system catalogs, here is not the place to prevent that). In particular, it seems just wrong to insist on an ownership check. That has the net effect of forcing people to use superuser for DROP OWNED BY, which surely is not an effect we want. Also there is no point in rebuilding the dependencies of the policy expressions, which aren't being changed. Lastly, locking the table also seems counterproductive; it's not helping to prevent race conditions, since we failed to re-read the pg_policy row after acquiring the lock. That means that concurrent DDL would likely result in "tuple concurrently updated/deleted" errors; which is the same behavior this code will produce, with less overhead. Per discussion of bug #17062. Back-patch to all supported versions, as the failure cases this eliminates seem just as undesirable in 9.6 as in HEAD. Discussion: https://postgr.es/m/1573181.1624220108@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/5a0f1c8c0193f0dd7fba50c22d96781fa2414007

  • Remove undesirable libpq dependency on stringinfo.c. Commit c0cb87fbb unwisely introduced a dependency on the StringInfo machinery in fe-connect.c. We must not use that in libpq, because it will do a summary exit(1) if it hits OOM, and that is not appropriate behavior for a general-purpose library. The goal of allowing arbitrary line lengths in service files doesn't seem like it's worth a lot of effort, so revert back to the previous method of using a stack-allocated buffer and failing on buffer overflow. This isn't an exact revert though. I kept that patch's refactoring to have a single exit path, as that seems cleaner than having each error path know what to do to clean up. Also, I made the fixed-size buffer 1024 bytes not 256, just to push off the need for an expandable buffer some more. There is more to do here; in particular the lack of any mechanical check for this type of mistake now seems pretty hazardous. But this fix gets us back to the level of robustness we had in v13, anyway. Discussion: https://postgr.es/m/daeb22ec6ca8ef61e94d766a9b35fb03cabed38e.camel@vmware.com https://git.postgresql.org/pg/commitdiff/8ec00dc5cd70e0e579e9fbf8661bc46f5ccd8078

  • Doc: update v14 release notes for revert of commit c0cb87fbb. https://git.postgresql.org/pg/commitdiff/dcffc9ba8a1e0ab1b0a57e9b9d38e3dc9960f83f

  • Remove memory leaks in isolationtester. specscanner.l leaked a kilobyte of memory per token of the spec file. Apparently somebody thought that the introductory code block would be executed once; but it's once per yylex() call. A couple of functions in isolationtester.c leaked small amounts of memory due to not bothering to free one-time allocations. Might as well improve these so that valgrind gives this program a clean bill of health. Also get rid of an ugly static variable. Coverity complained about one of the one-time leaks, which led me to try valgrind'ing isolationtester, which led to discovery of the larger leak. https://git.postgresql.org/pg/commitdiff/642c0697c96b9c6ba5d194653a329f7820565f01

Michaël Paquier pushed:

Bruce Momjian pushed:

Álvaro Herrera pushed:

Noah Misch pushed:

Amit Kapila pushed:

Alexander Korotkov pushed:

Peter Geoghegan pushed:

  • Remove unneeded field from VACUUM state. Bugfix commit 5fc89376 effectively made the lock_waiter_detected field from vacuumlazy.c's global state struct into private state owned by lazy_truncate_heap(). Finish this off by replacing the struct field with a local variable. https://git.postgresql.org/pg/commitdiff/958cfbcf2dd338e3179c2d8a35f48bde020eba60

  • Support disabling index bypassing by VACUUM. Generalize the INDEX_CLEANUP VACUUM parameter (and the corresponding reloption): make it into a ternary style boolean parameter. It now exposes a third option, "auto". The "auto" option (which is now the default) enables the "bypass index vacuuming" optimization added by commit 1e55e7d1. "VACUUM (INDEX_CLEANUP TRUE)" is redefined to once again make VACUUM simply do any required index vacuuming, regardless of how few dead tuples are encountered during the first scan of the target heap relation (unless there are exactly zero). This gives users a way of opting out of the "bypass index vacuuming" optimization, if for whatever reason that proves necessary. It is also expected to be used by PostgreSQL developers as a testing option from time to time. "VACUUM (INDEX_CLEANUP FALSE)" does the same thing as it always has: it forcibly disables both index vacuuming and index cleanup. It's not expected to be used much in PostgreSQL

  • The failsafe mechanism added by commit 1e55e7d1 addresses the same problem in a simpler way. INDEX_CLEANUP can now be thought of as a testing and compatibility option. Author: Peter Geoghegan pg@bowt.ie Reviewed-By: Masahiko Sawada sawada.mshk@gmail.com Reviewed-By: Justin Pryzby pryzby@telsasoft.com Discussion: https://postgr.es/m/CAH2-WznrBoCST4_Gxh_G9hA8NzGUbeBGnOUC8FcXcrhqsv6OHQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/3499df0dee8c4ea51d264a674df5b5e31991319a

  • Remove overzealous VACUUM failsafe assertions. The failsafe can trigger when index processing is already disabled. This can happen when VACUUM's INDEX_CLEANUP parameter is "off" and the failsafe happens to trigger. Remove assertions that assume that index processing is directly tied to the failsafe. Oversight in commit c242baa4, which made it possible for the failsafe to trigger in a two-pass strategy VACUUM that has yet to make its first call to lazy_vacuum_all_indexes(). https://git.postgresql.org/pg/commitdiff/e8f201ab82be234b2f103234cf9f262f9afeaeba

  • Add list of ignorable pgindent commits for git-blame. Add a .git-blame-ignore-revs file with a list of pgindent, pgperlyidy, and reformat-dat-files commit hashes. Postgres hackers that configure git to use the ignore file will get git-blame output that avoids attributing line changes to the ignored indent commits. This makes git-blame output much easier to work with in practice. Author: Peter Geoghegan pg@bowt.ie Reviewed-By: Tom Lane tgl@sss.pgh.pa.us Discussion: https://postgr.es/m/CAH2-Wz=cVh3GHTP6SdLU-Gnmt2zRdF8vZkcrFdSzXQ=WhbWm9Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/8e638845ff6bb255ad1dea15831089a234535391

Andrew Dunstan pushed:

Heikki Linnakangas pushed:

  • Fix outdated comment that talked about seek position of WAL file. Since commit c24dcd0cfd, we have been using pg_pread() to read the WAL file, which doesn't change the seek position (unless we fall back to the implementation in src/port/pread.c). Update comment accordingly. Backpatch-through: 12, where we started to use pg_pread() https://git.postgresql.org/pg/commitdiff/d0303bc8d2670d11c9df9220aa08a2c33529e100

  • Tidy up GetMultiXactIdMembers()'s behavior on error. One of the error paths left *members uninitialized. That's not a live bug, because most callers don't look at *members when the function returns -1, but let's be tidy. One caller, in heap_lock_tuple(), does "if (members != NULL) pfree(members)", but AFAICS it never passes an invalid 'multi' value so it should not reach that error case. The callers are also a bit inconsistent in their expectations. heap_lock_tuple() pfrees the 'members' array if it's not-NULL, others pfree() it if "nmembers >= 0", and others if "nmembers > 0". That's not a live bug either, because the function should never return 0, but add an Assert for that to make it more clear. I left the callers alone for now. I also moved the line where we set *nmembers. It wasn't wrong before, but I like to do that right next to the 'return' statement, to make it clear that it's always set on return. Also remove one unreachable return statement after ereport(ERROR), for brevity and for consistency with the similar if-block right after it. Author: Greg Nancarrow with the additional changes by me Backpatch-through: 9.6, all supported versions https://git.postgresql.org/pg/commitdiff/d24c5658a80c8f5037e9e1948de311d3f3350f12

  • Prevent race condition while reading relmapper file. Contrary to the comment here, POSIX does not guarantee atomicity of a read(), if another process calls write() concurrently. Or at least Linux does not. Add locking to load_relmap_file() to avoid the race condition. Fixes bug #17064. Thanks to Alexander Lakhin for the report and test case. Backpatch-through: 9.6, all supported versions. Discussion: https://www.postgresql.org/message-id/17064-bb0d7904ef72add3@postgresql.org https://git.postgresql.org/pg/commitdiff/b6d8d2073f228d9f7198f1f9826d8f6ab643c219

  • Another fix to relmapper race condition. In previous commit, I missed that relmap_redo() was also not acquiring the RelationMappingLock. Thanks to Thomas Munro for pointing that out. Backpatch-through: 9.6, like previous commit. Discussion: https://www.postgresql.org/message-id/CA%2BhUKGLev%3DPpOSaL3WRZgOvgk217et%2BbxeJcRr4eR-NttP1F6Q%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/9b8ed0f52bffceaccf6fa650ffe482e7da20fbf2

Tomáš Vondra pushed:

  • Fix copying data into slots with FDW batching. Commit b676ac443b optimized handling of tuple slots with bulk inserts into foreign tables, so that the slots are initialized only once and reused for all batches. The data was however copied into the slots only after the initialization, inserting duplicate values when the slot gets reused. Fixed by moving the ExecCopySlot outside the init branch. The existing postgres_fdw tests failed to catch this due to inserting data into foreign tables without unique indexes, and then checking only the number of inserted rows. This adds a new test with both a unique index and a check of inserted values. Reported-by: Alexander Pyhalov Discussion: https://postgr.es/m/7a8cf8d56b3d18e5c0bccd6cd42d04ac%40postgrespro.ru https://git.postgresql.org/pg/commitdiff/99cea49d6525e30bc3768e4ffb95377e7584dea1

Fujii Masao pushed:

Peter Eisentraut pushed:

David Rowley pushed:

  • Fix assert failure in expand_grouping_sets. linitial_node() fails in assert enabled builds if the given pointer is not of the specified type. Here the type is IntList. The code thought it should be expecting List, but it was wrong. In the existing tests which run this code the initial list element is always NIL. Since linitial_node() allows NULL, we didn't trigger any assert failures in the existing regression tests. There is still some discussion as to whether we need a few more tests in this area, but for now, since beta2 is looming, fix the bug first. Bug: #17067 Discussion: https://postgr.es/m/17067-665d50fa321f79e0@postgresql.org Reported-by: Yaoguang Chen https://git.postgresql.org/pg/commitdiff/8d29d45d9b3cab95a866efbcdd9138b3d76741b3

Andres Freund pushed:

  • Use correct horizon when vacuuming catalog relations. In dc7420c2c92 I (Andres) accidentally used RelationIsAccessibleInLogicalDecoding() as the sole condition to use the non-shared catalog horizon in GetOldestNonRemovableTransactionId(). That is incorrect, as RelationIsAccessibleInLogicalDecoding() checks whether wal_level is logical. The correct check, as done e.g. in GlobalVisTestFor(), is to check IsCatalogRelation() and RelationIsAccessibleInLogicalDecoding(). The observed misbehavior of this bug was that there could be an endless loop in lazy_scan_prune(), because the horizons used in heap_page_prune() and the individual tuple liveliness checks did not match. Likely there are other potential consequences as well. A later commit will unify the determination which horizon has to be used, and add additional assertions to make it easier to catch a bug like this. Reported-By: Justin Pryzby pryzby@telsasoft.com Diagnosed-By: Matthias van de Meent boekewurm+postgres@gmail.com Author: Matthias van de Meent boekewurm+postgres@gmail.com Discussion: https://postgr.es/m/CAEze2Wg32Y9+WJfw=aofkRx1ZRFt_Ev6bNPc4PSaz7PjSFtZgQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5a1e1d83022b976ebdec5cfa8f255c4278b75b8e

Joe Conway pushed:

Thomas Munro pushed:

Pending Patches

Nitin Jadhav sent in another revision of a patch to make a way to track the progress of the operations performed during the startup process.

Yuzuko Hosoya sent in a patch to control whether autoanalyze runs for partition operations via new GUCs autovacuum_analyze_attach_partition, autovacuum_analyze_detach_partition, and autovacuum_analyze_drop_partition.

Peter Eisentraut sent in a patch to add an index OID macro argument to DECLARE_INDEX. Instead of defining symbols such as AmOidIndexId explicitly, include them as an argument of DECLARE_INDEX() and have genbki.pl generate the way as the table OID symbols from the CATALOG() declaration

Filip Gospodinov sent in a patch to intended to fix a bug that manifested as the shipped pkg-config file is broken for statically linking libpq because libpgcommon and libpgport are missing. The fix adds those two missing private dependencies.

Heikki Linnakangas sent in another revision of a patch to split xlog.c into xlog.c and xlogrecovery.c. This moves the functions related to performing WAL recovery into the new xlogrecovery.c source file, leaving xlog.c responsible for maintaining the WAL buffers, coordinating the startup and switch from recovery to normal operations, and other miscellaneous stuff that have always been in xlog.c

Greg Nancarrow sent in two revisions of a patch to remove the useless int64 range checks on BIGINT sequence MINVALUE/MAXVALUE values.

Peter Geoghegan sent in three revisions of a patch to add a list of ignorable pgindent commits for "git blame".

David Rowley sent in another revision of a patch to add a new hash table type which has stable pointers, and use same to speed up SMgr.

Peter Smith and Ajin Cherian traded patches to add an option to support prepared transactions to built-in logical replication.

Daniel Gustafsson and Michaël Paquier traded patches to document some SSL/TLS related acronyms, and replace usage of SSL with SSL/TLS, the latter being more accurate and up to date.

Simon Riggs sent in three revisions of a patch to add a documentation chapter on hash indexes.

Atsushi Torikoshi sent in another revision of a patch to add a function to log the complete query string and its plan for the query currently running on the backend with the specified process ID.

Li Japin sent in a patch to make a minor code beautification in RelationGetIdentityKeyBitmap.

Bertrand Drouvot sent in another revision of a patch to enable minimal logical decoding on standbys.

Alexander Pyhalov sent in another revision of a patch to allow pushing CASE expressions to foreign servers.

Peter Eisentraut sent in a patch to Make the Unicode makefile more parallel-safe, Make UCS_to_most.pl process encodings in sorted order, remove some whitespace in generated C output to get it in line with the coding style of the rest of the project, simplify the code-generation code, and fix indentation in generated output.

Maxim Orlov sent in another revision of a patch to fix parallel worker failed assertion and coredump.

Vigneshwaran C sent in two more revisions of a patch to add schema level support for PUBLICATIONs.

Mike Fiedler sent in a patch to emit the namespace in post-copy output.

Emre Hasegeli sent in a patch to decouple operator classes from index access methods.

Jacob Champion sent in two revisions of a patch to decouple the SASL framework from the SCRAM code, as it may be useful separately.

Yugo Nagata sent in another revision of a patch to address some Pgbench errors by using the Variables structure for client variables. This is most important when it is used to reset client variables during the repeating of transactions after serialization/deadlock failures. Don't allocate Variable structs one by one. Instead, add a constant margin each time it overflows. Also included, a fix for pgbench errors and serialization/deadlock retries.

Jacob Champion, Michaël Paquier, and Tom Lane traded patches to make it possible to use jsonapi with libpq.

Gurjeet Singh sent in a patch to add automatic notification for top transaction IDs.

Haiying Tang sent in another revision of a patch to support tab completion with a query result for upper case inputs in psql.

Peter Eisentraut sent in a patch to make genbki error handling more useful by counting the errors and error out at the end instead of just writing warnings for invalid cross-catalog lookups.

Daniel Gustafsson sent in another revision of a patch to support NSS as a TLS backend for libpq.

Andrey V. Lepikhov sent in a patch to inform genericcostestimate()'s index selectivity value of the fact that in the case of an unique one-row btree index, scan only one row can be returned.

Tomáš Vondra sent in another revision of a patch to make it possible to replicate sequences in logical decoding.

Ranier Vilela sent in a patch to fix an uninitialized copy_data var in src/backend/commands/subscriptioncmds.c.

Peter Eisentraut sent in another revision of a patch to improve error messages about mismatching relkind by making the primary error message shorter and saying more directly that what was attempted is not possible.

Peter Eisentraut sent in a patch to add tests for UNBOUNDED syntax ambiguity.

Simon Riggs sent in a patch to make pgbench use COPY FREEZE.

Justin Pryzby sent in a patch to avoid double parentheses, and fix a comment in tablefunc.c to refer to the correct comment.

Craig Ringer sent in a patch to add some more PGDLLIMPORTs to expose some of the things heretofore unavailable on Windows.

Amit Kapila sent in a patch to allow streaming the changes after speculative aborts.

Hayato Kuroda sent in a patch to make ECPG's new DECLARE STATEMENT work for DEALLOCATE and DESCRIBE.

Bruce Momjian sent in another revision of a patch to implement cluster file encryption.

Jie Zhang sent in a patch to bring the generated sample postgresql.conf into line with what's actually the default for shared_buffers.

Richard Guo sent in a patch to use each rel as both outer and inner for anti joins.

Tom Lane sent in a patch to prevent any calls of abort() or exit() from inside libpq.

Alexander Korotkov sent in a patch to fix some small inconsistencies in the catalog definition of multirange operators.

Julien Rouhaud sent in another revision of a patch to preserve param location when generating a const node.

Andrey Borodin sent in another revision of a patch to reorganize the pglz compression code.

Julien Rouhaud and Ranier Vilela traded patches to add a pg_get_query_def() to deparse and print a rewritten SQL statement.

Tomáš Vondra sent in a PoC patch to do cardinality estimation using runtime sampling.