Info

With PM-2677 backported to v6.0, change stream pipelines containing the $changeStreamSplitLargeEvent stage generate V2 resume tokens by design. Customers should be able to add this new stage to their existing change stream pipelines and resume using the existing V1 resume tokens, which is v6.0's default. Currently, there is a bug in compareAgainstClientResumeToken() resulting in ChangeStreamFatalError: cannot resume stream; the resume token was not found.

Top User Comments

xgen-internal-githook commented on Thu, 9 Nov 2023 19:16:26 +0000: Author: {'name': 'Romans Kasperovics', 'email': 'romans.kasperovics@mongodb.com', 'username': 'romanskas'} Message: SERVER-81295 Fix v1/v2 change stream resume token transition Change stream implementation assumes change events are totally ordered by event '_id', which also serves as a resume token. In server versions prior to v7.0 the default resume token version is v1. Starting from v7.0 the default resume token version is v2, but the server still supports clients supplying v1 resume tokens (e.g., after server upgrade). The token version is a part of a token and all v1 tokens sort before v2 tokens. To ensure the correct event ordering, the server from v7.0 generates v1 tokens to be compared with the client's v1 token until the referred event is either located or surpassed. In this commit we fix the following issues: When resuming from the event within a transaction we need to consider 'txnOpIndex', because the whole transaction gets the same timestamp. Splitting large events is only supported with v2 tokens. If a resume token is a high-watermark v1 token, the next event sharing the same timestamp will have token version v1. If this event is over 16MB, splitting it will fail because of the token version. To fix this, we convert the client's high-watermark v1 token to v2. Branch: v6.0 https://github.com/mongodb/mongo/commit/4cf365118b715bca6461d741d4c1edecf1341817 xgen-internal-githook commented on Tue, 7 Nov 2023 11:10:02 +0000: Author: {'name': 'Romans Kasperovics', 'email': 'romans.kasperovics@mongodb.com', 'username': 'romanskas'} Message: SERVER-81295 Fix v1/v2 change stream resume token transition Change stream implementation assumes change events are totally ordered by event '_id', which also serves as a resume token. In server versions prior to v7.0 the default resume token version is v1. Starting from v7.0 the default resume token version is v2, but the server still supports clients supplying v1 resume tokens (e.g., after server upgrade). The token version is a part of a token and all v1 tokens sort before v2 tokens. To ensure the correct event ordering, the server from v7.0 generates v1 tokens to be compared with the client's v1 token until the referred event is either located or surpassed. In this commit we fix the following issues: When resuming from the event within a transaction we need to consider 'txnOpIndex', because the whole transaction gets the same timestamp. Splitting large events is only supported with v2 tokens. If a resume token is a high-watermark v1 token, the next event sharing the same timestamp will have token version v1. If this event is over 16MB, splitting it will fail because of the token version. To fix this, we convert the client's high-watermark v1 token to v2. Branch: v7.0 https://github.com/mongodb/mongo/commit/0a9c07cfa5449fa21a84dd5730bff6427b52db0b xgen-internal-githook commented on Fri, 3 Nov 2023 00:00:26 +0000: Author: {'name': 'Romans Kasperovics', 'email': 'romans.kasperovics@mongodb.com', 'username': 'romanskas'} Message: SERVER-81295 Fix v1/v2 change stream resume token transition Change stream implementation assumes change events are totally ordered by event '_id', which also serves as a resume token. In server versions prior to v7.0 the default resume token version is v1. Starting from v7.0 the default resume token version is v2, but the server still supports clients supplying v1 resume tokens (e.g., after server upgrade). The token version is a part of a token and all v1 tokens sort before v2 tokens. To ensure the correct event ordering, the server from v7.0 generates v1 tokens to be compared with the client's v1 token until the referred event is either located or surpassed. In this commit we fix the following issues: When resuming from the event within a transaction we need to consider 'txnOpIndex', because the whole transaction gets the same timestamp. Splitting large events is only supported with v2 tokens. If a resume token is a high-watermark v1 token, the next event sharing the same timestamp will have token version v1. If this event is over 16MB, splitting it will fail because of the token version. To fix this, we convert the client's high-watermark v1 token to v2. Branch: v7.1 https://github.com/mongodb/mongo/commit/7755281ad25f987d7cbc4e9feb5d09432796fe41 xgen-internal-githook commented on Sun, 29 Oct 2023 22:42:44 +0000: Author: {'name': 'Romans Kasperovics', 'email': 'romans.kasperovics@mongodb.com', 'username': 'romanskas'} Message: SERVER-81295 Fix v1/v2 change stream resume token transition Change stream implementation assumes change events are totally ordered by event '_id', which also serves as a resume token. In server versions prior to v7.0 the default resume token version is v1. Starting from v7.0 the default resume token version is v2, but the server still supports clients supplying v1 resume tokens (e.g., after server upgrade). The token version is a part of a token and all v1 tokens sort before v2 tokens. To ensure the correct event ordering, the server from v7.0 generates v1 tokens to be compared with the client's v1 token until the referred event is either located or surpassed. In this commit we fix the following issues: When resuming from the event within a transaction we need to consider 'txnOpIndex', because the whole transaction gets the same timestamp. Splitting large events is only supported with v2 tokens. If a resume token is a high-watermark v1 token, the next event sharing the same timestamp will have token version v1. If this event is over 16MB, splitting it will fail because of the token version. To fix this, we convert the client's high-watermark v1 token to v2. Branch: master https://github.com/mongodb/mongo/commit/24bb8e2f5be1cf103f7e2c5da524186e738cab5d

Steps to Reproduce