OPERATIONAL DEFECT DATABASE
...
...
Testing the bulkWrite DUP Key example from https://docs.mongodb.com/manual/reference/method/db.collection.bulkWrite/#bulk-write-operations in a transaction I can't find the writeErrors array in the result. Server: 4.0.4-ent WriteCommandError({ "operationTime" : Timestamp(1544613730, 3), "ok" : 0, "errmsg" : "E11000 duplicate key error collection: test.c index: _id_ dup key: { : 1.0 }", "code" : 11000, "codeName" : "DuplicateKey", "$clusterTime" : { "clusterTime" : Timestamp(1544613730, 3), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } }) Thinking that perhaps this is a client side issue I looked at the wire protocol response and I don't see it there either sudo tshark -nQ -i lo0 --enable-protocol "MONGO" -f "portrange 12108-12110" -Y "mongo.opcode==2013" -d "tcp.port==12108-12110,mongo" -t u -T json "mongo.msg.sections.section": { "mongo.msg.sections.section.kind": "0", "mongo.msg.sections.section.body": { "mongo.document.length": "268", "mongo.elements": { "mongo.element.name": "operationTime", "mongo.element.name_tree": { "mongo.element.type": "0x00000011", "mongo.element.value.int64": "6634065455302574083" }, "mongo.element.name": "ok", "mongo.element.name_tree": { "mongo.element.type": "0x00000001", "mongo.element.value.double": "0" }, "mongo.element.name": "errmsg", "mongo.element.name_tree": { "mongo.element.type": "0x00000002", "mongo.element.value.length": "77", "mongo.element.value.string": "E11000 duplicate key error collection: test.c index: _id_ dup key: { : 1.0 }" }, "mongo.element.name": "code", "mongo.element.name_tree": { "mongo.element.type": "0x00000010", "mongo.element.value.int": "11000" }, "mongo.element.name": "codeName", "mongo.element.name_tree": { "mongo.element.type": "0x00000002", "mongo.element.value.length": "13", "mongo.element.value.string": "DuplicateKey" }, "mongo.element.name": "$clusterTime", "mongo.element.name_tree": { "mongo.element.type": "0x00000003", "mongo.document": { "mongo.document.length": "88", "mongo.elements": { "mongo.element.name": "clusterTime", "mongo.element.name_tree": { "mongo.element.type": "0x00000011", "mongo.element.value.int64": "6634065455302574083" }, "mongo.element.name": "signature", "mongo.element.name_tree": { "mongo.element.type": "0x00000003", "mongo.document": { "mongo.document.length": "51", "mongo.elements": { "mongo.element.name": "hash", "mongo.element.name_tree": { "mongo.element.type": "0x00000005", "mongo.element.value.length": "20", "mongo.element.value.bytes": "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00" }, "mongo.element.name": "keyId", "mongo.element.name_tree": { "mongo.element.type": "0x00000012", "mongo.element.value.int64": "0" } } } } } } } } } } How do I identify failed items?
xgen-internal-githook commented on Tue, 12 Feb 2019 22:41:03 +0000: Author: {'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'} Message: SERVER-38583 Fix transaction insert writeError format Branch: master https://github.com/mongodb/mongo/commit/db3baf2ee165ebba924620b887814adefabf1e94 jesse commented on Wed, 9 Jan 2019 18:27:44 +0000: Yes, it is my plan to make multi-document transaction write errors include ok: 1. Drivers were written with the assumption that errors from bulk writes have the same structure whether in a transaction or not, so I'd like to make that assumption true. I see your point, that if the 50th document in a bulk insert fails, then in a transaction the previous 49 inserts were rolled back. But that's the same today if the application does writes one by one: session.startTransaction() collection.insertOne(session, {_id: 0}) // success collection.insertOne(session, {_id: 1}) // success collection.insertOne(session, {_id: 1}) // fails, aborts Here, too, it's the application's job to understand that the successful writes have been rolled back. So I think the goal should be to make bulk write error reporting the same, whether in a transaction or not. milkie commented on Wed, 9 Jan 2019 18:10:21 +0000: You didn't specify explicitly, but it is possible you are planning to make the multi-document transaction return ok:1 instead of ok:0? I don't think we can change the way the ok field is being presented here, since with non-transactional batch commands it is entirely possible that some writes succeeded and others did not (the failing ones then appearing in the writeErrors array), thus ok:1; whereas with a batch command inside a transaction, with any write failure none of the writes in the batch can succeed and they all always roll back, thus ok:0. Arguably, we could have made the batch commands return ok:0 if any write in the batch failed, but I believe that would be a disruptive semantic change to make at this point. jesse commented on Wed, 9 Jan 2019 17:26:04 +0000: Right, only inserts currently have difficulty determining which document failed in the batch, when part of a multi-document transaction. However, all write commands exhibit the same behavior: although they all have ok: 1 and include writeErrors if they fail outside a multi-document transaction, they have ok: 0 and omit writeErrors in a transaction. I think this is a bug, they should have the same error format whether they're in a multi-document transaction or not. I propose to update this storage API to indicate which record in the vector failed (the method will quit on the first error). Currently it's: virtual Status insertRecords(OperationContext* opCtx, std::vector* inOutRecords, const std::vector& timestamps) = 0; It will become: struct InsertRecordsResult { size_t nInserted; Status status; }; virtual InsertRecordsResult insertRecords(OperationContext* opCtx, std::vector* inOutRecords, const std::vector& timestamps) = 0; From there, I can update the error handling for all write commands in write_ops_exec.cpp to fix the error format for transactional writes. milkie commented on Wed, 9 Jan 2019 15:06:50 +0000: I believe only the insert command works this way; update and delete do not do this – is that correct? We could possibly mimic a "writeErrors" array for insert, but it would only ever have 1 element in it, since once you hit one error the transaction must be abandoned. Still, the one erroring element could be useful to users. jesse commented on Wed, 9 Jan 2019 14:47:11 +0000: Here's a regular error: > db.runCommand({insert: 'c', documents: [{_id: 0}, {_id: 0}]}) { "n" : 1, "writeErrors" : [ { "index" : 1, "code" : 11000, "keyPattern" : { "_id" : 1 }, "keyValue" : { "" : 0 }, "errmsg" : "E11000 duplicate key error collection: test.c index: _id_ dup key: { _id: 0.0 }" } ], "opTime" : { "ts" : Timestamp(1547044399, 3), "t" : NumberLong(6) }, "electionId" : ObjectId("7fffffff0000000000000006"), "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1547044399, 4), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1547044399, 3) } Versus an error in a transaction: > session.startTransaction(); session.getDatabase("test").runCommand({insert: 'c', documents: [{_id: 0}, {_id: 0}]}) { "operationTime" : Timestamp(1547044524, 1), "ok" : 0, "errmsg" : "E11000 duplicate key error collection: test.c index: _id_ dup key: { _id: 0.0 }", "code" : 11000, "codeName" : "DuplicateKey", "keyPattern" : { "_id" : 1 }, "keyValue" : { "" : 0 }, "$clusterTime" : { "clusterTime" : Timestamp(1547044524, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } } } The error in the transaction has ok: 0 and it lacks writeErrors, as well as opTime and electionId. The error info is moved to the top level of the reply. Theory: in insertBatchAndHandleErrors, outside a transaction, we try a bulk write two ways: first, try it all at once efficiently, and if that fails, try each operation within the bulk write individually, in order to determine which one(s) failed. Then serializeReply() assembles the reply with ok: 1 and writeErrors seen above. But if we are in a transaction, then insertBatchAndHandleErrors can't use that technique (we'd need nested transactions or savepoints in WiredTiger), so it throws an error without knowing which records failed. The exception bubbles up past where serializeReply() would have been called; the exception is handled much higher, perhaps in execCommandDatabase(), which marks the command ok: 0. craig.homa commented on Thu, 20 Dec 2018 16:19:13 +0000: Hey Repl, the Query team feels that it would be best for you to help with this. asya commented on Fri, 14 Dec 2018 16:21:04 +0000: Note that for 4.2 this message will be better because of SERVER-14801 - we will now show the field name (by showing index definition) and value that caused the duplicate exception. asya commented on Fri, 14 Dec 2018 16:17:15 +0000: Or are you saying you want to see the full document that caused the failure? asya commented on Fri, 14 Dec 2018 16:13:46 +0000: Can you clarify something for me? The error message contains the key value that caused the error, and transaction aborts on first error, so what information is lost? You won't ever have failed itemS only a single failed item, no?
Start a 3 node replset mlaunch init --replicaset --nodes 3 --binarypath $(m bin 4.0.4-ent) --port ${port} I'm using the following test code db.dropDatabase(); db.createCollection("c"); session = db.getMongo().startSession(); try { session.startTransaction(); coll = session.getDatabase("test").c; try { coll.bulkWrite([ { insertOne : { "document" : { _id: 1, v: 1} } }, { insertOne : { "document" : { _id: 2, v: 1} } }, { insertOne : { "document" : { _id: 1, v: 1} } } ]); } catch (error) { print("Operation exception, aborting txn..."); print(error.stack); session.abortTransaction(); throw error; } session.commitTransaction(); // Uses write concern set at transaction start. } catch (error) { print("Transaction exception"); print(error); } finally { session.endSession(); }
MongoDB Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
