providers/mana: fix lost CQ notification when re-arming without polling

When a CQ is re-armed with nothing polled since the last doorbell,
gdma_ring_cq_doorbell() advanced the reported consumer index by one so
the doorbell would differ from the previous one and take effect. On an
idle CQ that index runs past what the device has produced, so the device
rejects the whole doorbell including the arm bit, and the stale index
makes the next re-arm fail too. The CQ is left un-armed and completions
raise no completion-channel event.

Only claim the next index once the owner bits show the device produced
that CQE. If it has not, no notification can have fired, so the CQ is
still armed and needs no doorbell at all.

Fixes: 6dd8578d46e4 ("providers/mana: Use poll credit for arming of cqs")
Signed-off-by: Snehal Sanghvi <snsanghvi@microsoft.com>
Reviewed-by: Konstantin Taranov <kotaranov@microsoft.com>

providers/mana/doorbells.h

diff --git a/providers/mana/doorbells.h b/providers/mana/doorbells.h
index d805cdb..7651e76 100644
--- a/providers/mana/doorbells.h
+++ b/providers/mana/doorbells.h
@@ -87,6 +87,16 @@
 	mmio_flush_writes();
 }
 
+/* Has HW already produced the CQE at this index? Owner bits are written by HW,
+ * so a match proves idx < cur_cqe. Only meaningful for idx >= cq->head.
+ */
+static inline bool gdma_cq_idx_produced(struct mana_cq *cq, uint32_t idx)
+{
+	struct gdma_cqe *cqe = ((struct gdma_cqe *)cq->buf) + (idx % cq->cqe);
+
+	return cqe->owner_bits == ((idx / cq->cqe) & CQ_OWNER_MASK);
+}
+
 static inline void gdma_ring_cq_doorbell(struct mana_cq *cq, uint8_t arm)
 {
 	union gdma_doorbell_entry e;
@@ -94,8 +104,14 @@
 	uint32_t max_credit = cq->cqe << (GDMA_CQE_OWNER_BITS - 1);
 
 	if (cq->poll_credit >= max_credit) {
-		// To address the use-case of ibv that re-arms the CQ without polling
-		cq->poll_credit++;
+		// To address the use-case of ibv that re-arms the CQ without polling.
+		// (prod_idx + poll_credit - max_credit) is the index last given to HW,
+		// so only claim the next one once HW has produced it. Nothing produced
+		// implies no notification fired, so the CQ is still armed.
+		if (gdma_cq_idx_produced(cq, prod_idx + cq->poll_credit - max_credit))
+			cq->poll_credit++;
+		else
+			return;
 	} else {
 		// Set index of already polled CQE for unarm
 		cq->poll_credit = max_credit - (arm ? 0 : 1);