mem: Delay responses in the crossbar before forwarding

This patch changes how the crossbar classes deal with
responses. Instead of forwarding responses directly and burdening the
neighbouring modules in paying for the latency (through the
pkt->headerDelay), we now queue them before sending them.

The coherency protocol is not affected as requests and any snoop
requests/responses are still passed on in zero time. Thus, the
responses end up paying for any header delay accumulated when passing
through the crossbar. Any latency incurred on the request path will be
paid for on the response side, if no other module has dealt with it.

As a result of this patch, responses are returned at a later
point. This affects the number of outstanding transactions, and quite
a few regressions see an impact in blocking due to no MSHRs, increased
cache-miss latencies, etc.

Going forward we should be able to use the same concept also for snoop
responses, and any request that is not an express snoop.

1 files changed, 6 insertions, 8 deletions

diff --git a/src/mem/noncoherent_xbar.hh b/src/mem/noncoherent_xbar.hh
index 64a1064ab..ffcf9797f 100644
--- a/src/mem/noncoherent_xbar.hh
+++ b/src/mem/noncoherent_xbar.hh
@@ -84,18 +84,22 @@ class NoncoherentXBar : public BaseXBar
      * will be instantiated for each of the master ports connecting to
      * the crossbar.
      */
-    class NoncoherentXBarSlavePort : public SlavePort
+    class NoncoherentXBarSlavePort : public QueuedSlavePort
     {
       private:
 
         /** A reference to the crossbar to which this port belongs. */
         NoncoherentXBar &xbar;
 
+        /** A normal packet queue used to store responses. */
+        RespPacketQueue queue;
+
       public:
 
         NoncoherentXBarSlavePort(const std::string &_name,
                                 NoncoherentXBar &_xbar, PortID _id)
-            : SlavePort(_name, &_xbar, _id), xbar(_xbar)
+            : QueuedSlavePort(_name, &_xbar, queue, _id), xbar(_xbar),
+              queue(_xbar, *this)
         { }
 
       protected:
@@ -119,12 +123,6 @@ class NoncoherentXBar : public BaseXBar
         { xbar.recvFunctional(pkt, id); }
 
         /**
-         * When receiving a retry, pass it to the crossbar.
-         */
-        virtual void recvRespRetry()
-        { panic("Crossbar slave ports should never retry.\n"); }
-
-        /**
          * Return the union of all adress ranges seen by this crossbar.
          */
         virtual AddrRangeList getAddrRanges() const