syscall_emul: [patch 13/22] add system call retry capability

This changeset adds functionality that allows system calls to retry without
affecting thread context state such as the program counter or register values
for the associated thread context (when system calls return with a retry
fault).

This functionality is needed to solve problems with blocking system calls
in multi-process or multi-threaded simulations where information is passed
between processes/threads. Blocking system calls can cause deadlock because
the simulator itself is single threaded. There is only a single thread
servicing the event queue which can cause deadlock if the thread hits a
blocking system call instruction.

To illustrate the problem, consider two processes using the producer/consumer
sharing model. The processes can use file descriptors and the read and write
calls to pass information to one another. If the consumer calls the blocking
read system call before the producer has produced anything, the call will
block the event queue (while executing the system call instruction) and
deadlock the simulation.

The solution implemented in this changeset is to recognize that the system
calls will block and then generate a special retry fault. The fault will
be sent back up through the function call chain until it is exposed to the
cpu model's pipeline where the fault becomes visible. The fault will trigger
the cpu model to replay the instruction at a future tick where the call has
a chance to succeed without actually going into a blocking state.

In subsequent patches, we recognize that a syscall will block by calling a
non-blocking poll (from inside the system call implementation) and checking
for events. When events show up during the poll, it signifies that the call
would not have blocked and the syscall is allowed to proceed (calling an
underlying host system call if necessary). If no events are returned from the
poll, we generate the fault and try the instruction for the thread context
at a distant tick. Note that retrying every tick is not efficient.

As an aside, the simulator has some multi-threading support for the event
queue, but it is not used by default and needs work. Even if the event queue
was completely multi-threaded, meaning that there is a hardware thread on
the host servicing a single simulator thread contexts with a 1:1 mapping
between them, it's still possible to run into deadlock due to the event queue
barriers on quantum boundaries. The solution of replaying at a later tick
is the simplest solution and solves the problem generally.

41 files changed, 130 insertions, 65 deletions

diff --git a/src/arch/alpha/isa/decoder.isa b/src/arch/alpha/isa/decoder.isa
index d6de363a7..8789fa905 100644
--- a/src/arch/alpha/isa/decoder.isa
+++ b/src/arch/alpha/isa/decoder.isa
@@ -840,7 +840,7 @@ decode OPCODE default Unknown::unknown() {
                     exitSimLoop("halt instruction encountered");
                 }}, IsNonSpeculative);
                 0x83: callsys({{
-                    xc->syscall(R0);
+                    xc->syscall(R0, &fault);
                 }}, IsSerializeAfter, IsNonSpeculative, IsSyscall);
                 // Read uniq reg into ABI return value register (r0)
                 0x9e: rduniq({{ R0 = Runiq; }}, IsIprAccess);
diff --git a/src/arch/arm/faults.cc b/src/arch/arm/faults.cc
index b0d7700d1..740d71d02 100644
--- a/src/arch/arm/faults.cc
+++ b/src/arch/arm/faults.cc
@@ -784,7 +784,8 @@ SupervisorCall::invoke(ThreadContext *tc, const StaticInstPtr &inst)
         callNum = tc->readIntReg(INTREG_X8);
     else
         callNum = tc->readIntReg(INTREG_R7);
-    tc->syscall(callNum);
+    Fault fault;
+    tc->syscall(callNum, &fault);
 
     // Advance the PC since that won't happen automatically.
     PCState pc = tc->pcState();
diff --git a/src/arch/mips/isa/decoder.isa b/src/arch/mips/isa/decoder.isa
index 1f930f3f5..5c3c6f6b1 100644
--- a/src/arch/mips/isa/decoder.isa
+++ b/src/arch/mips/isa/decoder.isa
@@ -164,7 +164,7 @@ decode OPCODE_HI default Unknown::unknown() {
                     0x2: movz({{ Rd = (Rt == 0) ? Rs : Rd; }});
                     0x3: movn({{ Rd = (Rt != 0) ? Rs : Rd; }});
                     0x4: decode FullSystemInt {
-                        0: syscall_se({{ xc->syscall(R2); }},
+                        0: syscall_se({{ xc->syscall(R2, &fault); }},
                                 IsSerializeAfter, IsNonSpeculative);
                       default: syscall({{ fault = std::make_shared<SystemCallFault>(); }});
                     }
diff --git a/src/arch/power/isa/decoder.isa b/src/arch/power/isa/decoder.isa
index 30002fe33..71ef95b06 100644
--- a/src/arch/power/isa/decoder.isa
+++ b/src/arch/power/isa/decoder.isa
@@ -512,7 +512,7 @@ decode OPCODE default Unknown::unknown() {
         55: stfdu({{ Mem_df = Fs; }});
     }
 
-    17: IntOp::sc({{ xc->syscall(R0); }},
+    17: IntOp::sc({{ xc->syscall(R0, &fault); }},
                   [ IsSyscall, IsNonSpeculative, IsSerializeAfter ]);
 
     format FloatArithOp {
diff --git a/src/arch/riscv/faults.cc b/src/arch/riscv/faults.cc
index f5ba5c798..58baa4e32 100644
--- a/src/arch/riscv/faults.cc
+++ b/src/arch/riscv/faults.cc
@@ -87,5 +87,6 @@ BreakpointFault::invoke_se(ThreadContext *tc, const StaticInstPtr &inst)
 void
 SyscallFault::invoke_se(ThreadContext *tc, const StaticInstPtr &inst)
 {
-    tc->syscall(tc->readIntReg(SyscallNumReg));
+    Fault *fault = NoFault;
+    tc->syscall(tc->readIntReg(SyscallNumReg), fault);
 }
diff --git a/src/arch/sparc/faults.cc b/src/arch/sparc/faults.cc
index 6c3b35c9a..c09bd0da2 100644
--- a/src/arch/sparc/faults.cc
+++ b/src/arch/sparc/faults.cc
@@ -811,7 +811,8 @@ TrapInstruction::invoke(ThreadContext *tc, const StaticInstPtr &inst)
     SparcProcess *sp = dynamic_cast<SparcProcess *>(p);
     assert(sp);
 
-    sp->handleTrap(_n, tc);
+    Fault fault;
+    sp->handleTrap(_n, tc, &fault);
 
     // We need to explicitly advance the pc, since that's not done for us
     // on a faulting instruction
diff --git a/src/arch/sparc/linux/process.cc b/src/arch/sparc/linux/process.cc
index 0f72c69c7..d12f13048 100644
--- a/src/arch/sparc/linux/process.cc
+++ b/src/arch/sparc/linux/process.cc
@@ -65,14 +65,15 @@ Sparc32LinuxProcess::Sparc32LinuxProcess(ProcessParams * params,
     : Sparc32Process(params, objFile)
 {}
 
-void Sparc32LinuxProcess::handleTrap(int trapNum, ThreadContext *tc)
+void Sparc32LinuxProcess::handleTrap(int trapNum, ThreadContext *tc,
+                                     Fault *fault)
 {
     switch (trapNum) {
       case 0x10: //Linux 32 bit syscall trap
-        tc->syscall(tc->readIntReg(1));
+        tc->syscall(tc->readIntReg(1), fault);
         break;
       default:
-        SparcProcess::handleTrap(trapNum, tc);
+        SparcProcess::handleTrap(trapNum, tc, fault);
     }
 }
 
@@ -81,14 +82,15 @@ Sparc64LinuxProcess::Sparc64LinuxProcess(ProcessParams * params,
     : Sparc64Process(params, objFile)
 {}
 
-void Sparc64LinuxProcess::handleTrap(int trapNum, ThreadContext *tc)
+void Sparc64LinuxProcess::handleTrap(int trapNum, ThreadContext *tc,
+                                     Fault *fault)
 {
     switch (trapNum) {
       // case 0x10: // Linux 32 bit syscall trap
       case 0x6d: // Linux 64 bit syscall trap
-        tc->syscall(tc->readIntReg(1));
+        tc->syscall(tc->readIntReg(1), fault);
         break;
       default:
-        SparcProcess::handleTrap(trapNum, tc);
+        SparcProcess::handleTrap(trapNum, tc, fault);
     }
 }
diff --git a/src/arch/sparc/linux/process.hh b/src/arch/sparc/linux/process.hh
index 1c38576fd..778af1fda 100644
--- a/src/arch/sparc/linux/process.hh
+++ b/src/arch/sparc/linux/process.hh
@@ -70,7 +70,7 @@ class Sparc32LinuxProcess : public SparcLinuxProcess, public Sparc32Process
         return SparcLinuxProcess::getDesc32(callnum);
     }
 
-    void handleTrap(int trapNum, ThreadContext *tc);
+    void handleTrap(int trapNum, ThreadContext *tc, Fault *fault);
 };
 
 /// A process with emulated 32 bit SPARC/Linux syscalls.
@@ -86,7 +86,7 @@ class Sparc64LinuxProcess : public SparcLinuxProcess, public Sparc64Process
         return SparcLinuxProcess::getDesc(callnum);
     }
 
-    void handleTrap(int trapNum, ThreadContext *tc);
+    void handleTrap(int trapNum, ThreadContext *tc, Fault *fault);
 };
 
 SyscallReturn getresuidFunc(SyscallDesc *desc, int num,
diff --git a/src/arch/sparc/process.cc b/src/arch/sparc/process.cc
index 5c4f43b67..a8359a9ce 100644
--- a/src/arch/sparc/process.cc
+++ b/src/arch/sparc/process.cc
@@ -71,7 +71,7 @@ SparcProcess::SparcProcess(ProcessParams * params, ObjectFile *objFile,
 }
 
 void
-SparcProcess::handleTrap(int trapNum, ThreadContext *tc)
+SparcProcess::handleTrap(int trapNum, ThreadContext *tc, Fault *fault)
 {
     PCState pc = tc->pcState();
     switch (trapNum) {
diff --git a/src/arch/sparc/process.hh b/src/arch/sparc/process.hh
index 69ea6b41f..efdc0f443 100644
--- a/src/arch/sparc/process.hh
+++ b/src/arch/sparc/process.hh
@@ -61,7 +61,7 @@ class SparcProcess : public Process
   public:
 
     // Handles traps which request services from the operating system
-    virtual void handleTrap(int trapNum, ThreadContext *tc);
+    virtual void handleTrap(int trapNum, ThreadContext *tc, Fault *fault);
 
     Addr readFillStart() { return fillStart; }
     Addr readSpillStart() { return spillStart; }
diff --git a/src/arch/x86/isa/decoder/one_byte_opcodes.isa b/src/arch/x86/isa/decoder/one_byte_opcodes.isa
index 859d1f1b4..95bc7a5c1 100644
--- a/src/arch/x86/isa/decoder/one_byte_opcodes.isa
+++ b/src/arch/x86/isa/decoder/one_byte_opcodes.isa
@@ -400,8 +400,9 @@
                         // will sign extend it, and there's no easy way to
                         // specify only checking the first byte.
                         0xffffffffffffff80:
-                            SyscallInst::int80('xc->syscall(Rax)',
-                                 IsSyscall, IsNonSpeculative, IsSerializeAfter);
+                            SyscallInst::int80('xc->syscall(Rax, &fault)',
+                                               IsSyscall, IsNonSpeculative,
+                                               IsSerializeAfter);
                     }
 
                     default: Inst::INT(Ib);
diff --git a/src/arch/x86/isa/decoder/two_byte_opcodes.isa b/src/arch/x86/isa/decoder/two_byte_opcodes.isa
index 772177d42..97c3dd688 100644
--- a/src/arch/x86/isa/decoder/two_byte_opcodes.isa
+++ b/src/arch/x86/isa/decoder/two_byte_opcodes.isa
@@ -235,8 +235,9 @@
                 }
             }
             0x05: decode FullSystemInt {
-                0: SyscallInst::syscall('xc->syscall(Rax)',
-                        IsSyscall, IsNonSpeculative, IsSerializeAfter);
+                0: SyscallInst::syscall('xc->syscall(Rax, &fault)',
+                                        IsSyscall, IsNonSpeculative,
+                                        IsSerializeAfter);
                 default: decode MODE_MODE {
                     0x0: decode MODE_SUBMODE {
                         0x0: Inst::SYSCALL_64();
@@ -422,8 +423,9 @@
             0x2: Inst::RDMSR();
             0x3: rdpmc();
             0x4: decode FullSystemInt {
-                0: SyscallInst::sysenter('xc->syscall(Rax)',
-                        IsSyscall, IsNonSpeculative, IsSerializeAfter);
+                0: SyscallInst::sysenter('xc->syscall(Rax, &fault)',
+                                         IsSyscall, IsNonSpeculative,
+                                         IsSerializeAfter);
                 default: sysenter();
             }
             0x5: sysexit();
diff --git a/src/arch/x86/process.cc b/src/arch/x86/process.cc
index dfbd41e4e..c1e4f710a 100644
--- a/src/arch/x86/process.cc
+++ b/src/arch/x86/process.cc
@@ -134,7 +134,7 @@ X86_64Process::X86_64Process(ProcessParams *params, ObjectFile *objFile,
 }
 
 void
-I386Process::syscall(int64_t callnum, ThreadContext *tc)
+I386Process::syscall(int64_t callnum, ThreadContext *tc, Fault *fault)
 {
     TheISA::PCState pc = tc->pcState();
     Addr eip = pc.pc();
@@ -143,7 +143,7 @@ I386Process::syscall(int64_t callnum, ThreadContext *tc)
         pc.npc(vsyscallPage.base + vsyscallPage.vsysexitOffset);
         tc->pcState(pc);
     }
-    X86Process::syscall(callnum, tc);
+    X86Process::syscall(callnum, tc, fault);
 }
 
 
diff --git a/src/arch/x86/process.hh b/src/arch/x86/process.hh
index fa95b4ff4..ef0329329 100644
--- a/src/arch/x86/process.hh
+++ b/src/arch/x86/process.hh
@@ -130,7 +130,7 @@ namespace X86ISA
         void argsInit(int intSize, int pageSize);
         void initState();
 
-        void syscall(int64_t callnum, ThreadContext *tc);
+        void syscall(int64_t callnum, ThreadContext *tc, Fault *fault);
         X86ISA::IntReg getSyscallArg(ThreadContext *tc, int &i);
         X86ISA::IntReg getSyscallArg(ThreadContext *tc, int &i, int width);
         void setSyscallArg(ThreadContext *tc, int i, X86ISA::IntReg val);
diff --git a/src/arch/x86/pseudo_inst.cc b/src/arch/x86/pseudo_inst.cc
index acf310631..c0ec11059 100644
--- a/src/arch/x86/pseudo_inst.cc
+++ b/src/arch/x86/pseudo_inst.cc
@@ -49,7 +49,9 @@ m5Syscall(ThreadContext *tc)
 {
     DPRINTF(PseudoInst, "PseudoInst::m5Syscall()\n");
 
-    tc->syscall(tc->readIntReg(INTREG_RAX));
+    Fault fault;
+    tc->syscall(tc->readIntReg(INTREG_RAX), &fault);
+
     MiscReg rflags = tc->readMiscReg(MISCREG_RFLAGS);
     rflags &= ~(1 << 16);
     tc->setMiscReg(MISCREG_RFLAGS, rflags);
diff --git a/src/cpu/BaseCPU.py b/src/cpu/BaseCPU.py
index c85e5afda..7b8a615ea 100644
--- a/src/cpu/BaseCPU.py
+++ b/src/cpu/BaseCPU.py
@@ -142,6 +142,8 @@ class BaseCPU(MemObject):
 
     checker = Param.BaseCPU(NULL, "checker CPU")
 
+    syscallRetryLatency = Param.Cycles(10000, "Cycles to wait until retry")
+
     do_checkpoint_insts = Param.Bool(True,
         "enable checkpoint pseudo instructions")
     do_statistics_insts = Param.Bool(True,
diff --git a/src/cpu/base.cc b/src/cpu/base.cc
index 10b8ce297..08f95ea49 100644
--- a/src/cpu/base.cc
+++ b/src/cpu/base.cc
@@ -135,7 +135,8 @@ BaseCPU::BaseCPU(Params *p, bool is_checker)
       numThreads(p->numThreads), system(p->system),
       functionTraceStream(nullptr), currentFunctionStart(0),
       currentFunctionEnd(0), functionEntryTick(0),
-      addressMonitor(p->numThreads)
+      addressMonitor(p->numThreads),
+      syscallRetryLatency(p->syscallRetryLatency)
 {
     // if Python did not provide a valid ID, do it here
     if (_cpuId == -1 ) {
diff --git a/src/cpu/base.hh b/src/cpu/base.hh
index 6622339e0..14dfc260b 100644
--- a/src/cpu/base.hh
+++ b/src/cpu/base.hh
@@ -588,6 +588,8 @@ class BaseCPU : public MemObject
         assert(tid < numThreads);
         return &addressMonitor[tid];
     }
+
+    Cycles syscallRetryLatency;
 };
 
 #endif // THE_ISA == NULL_ISA
diff --git a/src/cpu/checker/cpu.hh b/src/cpu/checker/cpu.hh
index 21ff9c7f7..e47c88484 100644
--- a/src/cpu/checker/cpu.hh
+++ b/src/cpu/checker/cpu.hh
@@ -393,7 +393,7 @@ class CheckerCPU : public BaseCPU, public ExecContext
     void wakeup(ThreadID tid) override { }
     // Assume that the normal CPU's call to syscall was successful.
     // The checker's state would have already been updated by the syscall.
-    void syscall(int64_t callnum) override { }
+    void syscall(int64_t callnum, Fault *fault) override { }
 
     void handleError()
     {
diff --git a/src/cpu/checker/thread_context.hh b/src/cpu/checker/thread_context.hh
index 5fcb82f6d..0313d079b 100644
--- a/src/cpu/checker/thread_context.hh
+++ b/src/cpu/checker/thread_context.hh
@@ -146,8 +146,8 @@ class CheckerThreadContext : public ThreadContext
     SETranslatingPortProxy &getMemProxy() { return actualTC->getMemProxy(); }
 
     /** Executes a syscall in SE mode. */
-    void syscall(int64_t callnum)
-    { return actualTC->syscall(callnum); }
+    void syscall(int64_t callnum, Fault *fault)
+    { return actualTC->syscall(callnum, fault); }
 
     Status status() const { return actualTC->status(); }
 
diff --git a/src/cpu/exec_context.hh b/src/cpu/exec_context.hh
index dd718b56a..b21f0767a 100644
--- a/src/cpu/exec_context.hh
+++ b/src/cpu/exec_context.hh
@@ -228,7 +228,7 @@ class ExecContext {
     /**
      * Executes a syscall specified by the callnum.
      */
-    virtual void syscall(int64_t callnum) = 0;
+    virtual void syscall(int64_t callnum, Fault *fault) = 0;
 
     /** @} */
 
diff --git a/src/cpu/minor/exec_context.hh b/src/cpu/minor/exec_context.hh
index 6235721cf..6b2eae0f1 100644
--- a/src/cpu/minor/exec_context.hh
+++ b/src/cpu/minor/exec_context.hh
@@ -241,12 +241,12 @@ class ExecContext : public ::ExecContext
     }
 
     void
-    syscall(int64_t callnum) override
+    syscall(int64_t callnum, Fault *fault) override
      {
         if (FullSystem)
             panic("Syscall emulation isn't available in FS mode.\n");
 
-        thread.syscall(callnum);
+        thread.syscall(callnum, fault);
     }
 
     ThreadContext *tcBase() override { return thread.getTC(); }
diff --git a/src/cpu/o3/commit.hh b/src/cpu/o3/commit.hh
index 48c169389..3cce7f69c 100644
--- a/src/cpu/o3/commit.hh
+++ b/src/cpu/o3/commit.hh
@@ -235,7 +235,7 @@ class DefaultCommit
     size_t numROBFreeEntries(ThreadID tid);
 
     /** Generates an event to schedule a squash due to a trap. */
-    void generateTrapEvent(ThreadID tid);
+    void generateTrapEvent(ThreadID tid, Fault inst_fault);
 
     /** Records that commit needs to initiate a squash due to an
      * external state update through the TC.
diff --git a/src/cpu/o3/commit_impl.hh b/src/cpu/o3/commit_impl.hh
index c6c6ea723..ea77f18fb 100644
--- a/src/cpu/o3/commit_impl.hh
+++ b/src/cpu/o3/commit_impl.hh
@@ -526,13 +526,16 @@ DefaultCommit<Impl>::numROBFreeEntries(ThreadID tid)
 
 template <class Impl>
 void
-DefaultCommit<Impl>::generateTrapEvent(ThreadID tid)
+DefaultCommit<Impl>::generateTrapEvent(ThreadID tid, Fault inst_fault)
 {
     DPRINTF(Commit, "Generating trap event for [tid:%i]\n", tid);
 
     TrapEvent *trap = new TrapEvent(this, tid);
 
-    cpu->schedule(trap, cpu->clockEdge(trapLatency));
+    Cycles latency = dynamic_pointer_cast<SyscallRetryFault>(inst_fault) ?
+                     cpu->syscallRetryLatency : trapLatency;
+
+    cpu->schedule(trap, cpu->clockEdge(latency));
     trapInFlight[tid] = true;
     thread[tid]->trapPending = true;
 }
@@ -767,10 +770,11 @@ DefaultCommit<Impl>::handleInterrupt()
 
         commitStatus[0] = TrapPending;
 
+        interrupt = NoFault;
+
         // Generate trap squash event.
-        generateTrapEvent(0);
+        generateTrapEvent(0, interrupt);
 
-        interrupt = NoFault;
         avoidQuiesceLiveLock = false;
     } else {
         DPRINTF(Commit, "Interrupt pending: instruction is %sin "
@@ -1240,7 +1244,7 @@ DefaultCommit<Impl>::commitHead(DynInstPtr &head_inst, unsigned inst_num)
         }
 
         // Generate trap squash event.
-        generateTrapEvent(tid);
+        generateTrapEvent(tid, inst_fault);
         return false;
     }
 
diff --git a/src/cpu/o3/cpu.cc b/src/cpu/o3/cpu.cc
index d85895030..8d38ed1f2 100644
--- a/src/cpu/o3/cpu.cc
+++ b/src/cpu/o3/cpu.cc
@@ -972,7 +972,7 @@ FullO3CPU<Impl>::trap(const Fault &fault, ThreadID tid,
 
 template <class Impl>
 void
-FullO3CPU<Impl>::syscall(int64_t callnum, ThreadID tid)
+FullO3CPU<Impl>::syscall(int64_t callnum, ThreadID tid, Fault *fault)
 {
     DPRINTF(O3CPU, "[tid:%i] Executing syscall().\n\n", tid);
 
@@ -983,7 +983,7 @@ FullO3CPU<Impl>::syscall(int64_t callnum, ThreadID tid)
     ++(this->thread[tid]->funcExeInst);
 
     // Execute the actual syscall.
-    this->thread[tid]->syscall(callnum);
+    this->thread[tid]->syscall(callnum, fault);
 
     // Decrease funcExeInst by one as the normal commit will handle
     // incrementing it.
diff --git a/src/cpu/o3/cpu.hh b/src/cpu/o3/cpu.hh
index 2065202f7..abe036b09 100644
--- a/src/cpu/o3/cpu.hh
+++ b/src/cpu/o3/cpu.hh
@@ -344,7 +344,7 @@ class FullO3CPU : public BaseO3CPU
     /** Executes a syscall.
      * @todo: Determine if this needs to be virtual.
      */
-    void syscall(int64_t callnum, ThreadID tid);
+    void syscall(int64_t callnum, ThreadID tid, Fault *fault);
 
     /** Starts draining the CPU's pipeline of all instructions in
      * order to stop all memory accesses. */
diff --git a/src/cpu/o3/dyn_inst.hh b/src/cpu/o3/dyn_inst.hh
index 6740c601d..8ab9979d2 100644
--- a/src/cpu/o3/dyn_inst.hh
+++ b/src/cpu/o3/dyn_inst.hh
@@ -237,7 +237,7 @@ class BaseO3DynInst : public BaseDynInst<Impl>
     bool simPalCheck(int palFunc);
 
     /** Emulates a syscall. */
-    void syscall(int64_t callnum);
+    void syscall(int64_t callnum, Fault *fault);
 
   public:
 
diff --git a/src/cpu/o3/dyn_inst_impl.hh b/src/cpu/o3/dyn_inst_impl.hh
index 06c0e15f3..00bcb3345 100644
--- a/src/cpu/o3/dyn_inst_impl.hh
+++ b/src/cpu/o3/dyn_inst_impl.hh
@@ -242,7 +242,7 @@ BaseO3DynInst<Impl>::simPalCheck(int palFunc)
 
 template <class Impl>
 void
-BaseO3DynInst<Impl>::syscall(int64_t callnum)
+BaseO3DynInst<Impl>::syscall(int64_t callnum, Fault *fault)
 {
     if (FullSystem)
         panic("Syscall emulation isn't available in FS mode.\n");
@@ -251,7 +251,7 @@ BaseO3DynInst<Impl>::syscall(int64_t callnum)
     // changes, update this instruction's nextPC because the syscall
     // must have changed the nextPC.
     TheISA::PCState curPC = this->cpu->pcState(this->threadNumber);
-    this->cpu->syscall(callnum, this->threadNumber);
+    this->cpu->syscall(callnum, this->threadNumber, fault);
     TheISA::PCState newPC = this->cpu->pcState(this->threadNumber);
     if (!(curPC == newPC)) {
         this->pcState(newPC);
diff --git a/src/cpu/o3/thread_context.hh b/src/cpu/o3/thread_context.hh
index 87b7d9198..0321f57f7 100755
--- a/src/cpu/o3/thread_context.hh
+++ b/src/cpu/o3/thread_context.hh
@@ -258,8 +258,8 @@ class O3ThreadContext : public ThreadContext
     { thread->storeCondFailures = sc_failures; }
 
     /** Executes a syscall in SE mode. */
-    virtual void syscall(int64_t callnum)
-    { return cpu->syscall(callnum, thread->threadId()); }
+    virtual void syscall(int64_t callnum, Fault *fault)
+    { return cpu->syscall(callnum, thread->threadId(), fault); }
 
     /** Reads the funcExeInst counter. */
     virtual Counter readFuncExeInst() { return thread->funcExeInst; }
diff --git a/src/cpu/o3/thread_state.hh b/src/cpu/o3/thread_state.hh
index 7765f86ea..4b4f51e8f 100644
--- a/src/cpu/o3/thread_state.hh
+++ b/src/cpu/o3/thread_state.hh
@@ -140,7 +140,10 @@ struct O3ThreadState : public ThreadState {
     ThreadContext *getTC() { return tc; }
 
     /** Handles the syscall. */
-    void syscall(int64_t callnum) { process->syscall(callnum, tc); }
+    void syscall(int64_t callnum, Fault *fault)
+    {
+        process->syscall(callnum, tc, fault);
+    }
 
     void dumpFuncProfile()
     {
diff --git a/src/cpu/simple/atomic.cc b/src/cpu/simple/atomic.cc
index c09f0c526..6c31f1ddd 100644
--- a/src/cpu/simple/atomic.cc
+++ b/src/cpu/simple/atomic.cc
@@ -628,6 +628,7 @@ AtomicSimpleCPU::tick()
 
             preExecute();
 
+            Tick stall_ticks = 0;
             if (curStaticInst) {
                 fault = curStaticInst->execute(&t_info, traceData);
 
@@ -641,6 +642,13 @@ AtomicSimpleCPU::tick()
                     traceData = NULL;
                 }
 
+                if (dynamic_pointer_cast<SyscallRetryFault>(fault)) {
+                    // Retry execution of system calls after a delay.
+                    // Prevents immediate re-execution since conditions which
+                    // caused the retry are unlikely to change every tick.
+                    stall_ticks += clockEdge(syscallRetryLatency) - curTick();
+                }
+
                 postExecute();
             }
 
@@ -649,7 +657,6 @@ AtomicSimpleCPU::tick()
                         curStaticInst->isFirstMicroop()))
                 instCnt++;
 
-            Tick stall_ticks = 0;
             if (simulate_inst_stalls && icache_access)
                 stall_ticks += icache_latency;
 
diff --git a/src/cpu/simple/exec_context.hh b/src/cpu/simple/exec_context.hh
index 430790c09..bfae118ad 100644
--- a/src/cpu/simple/exec_context.hh
+++ b/src/cpu/simple/exec_context.hh
@@ -323,12 +323,12 @@ class SimpleExecContext : public ExecContext {
     /**
      * Executes a syscall specified by the callnum.
      */
-    void syscall(int64_t callnum) override
+    void syscall(int64_t callnum, Fault *fault) override
     {
         if (FullSystem)
             panic("Syscall emulation isn't available in FS mode.");
 
-        thread->syscall(callnum);
+        thread->syscall(callnum, fault);
     }
 
     /** Returns a pointer to the ThreadContext. */
diff --git a/src/cpu/simple/timing.cc b/src/cpu/simple/timing.cc
index 43c50b948..1c468dc99 100644
--- a/src/cpu/simple/timing.cc
+++ b/src/cpu/simple/timing.cc
@@ -670,9 +670,15 @@ TimingSimpleCPU::advanceInst(const Fault &fault)
         return;
 
     if (fault != NoFault) {
-        advancePC(fault);
         DPRINTF(SimpleCPU, "Fault occured, scheduling fetch event\n");
-        reschedule(fetchEvent, clockEdge(), true);
+
+        advancePC(fault);
+
+        Tick stall = dynamic_pointer_cast<SyscallRetryFault>(fault) ?
+                     clockEdge(syscallRetryLatency) : clockEdge();
+
+        reschedule(fetchEvent, stall, true);
+
         _status = Faulting;
         return;
     }
diff --git a/src/cpu/simple_thread.hh b/src/cpu/simple_thread.hh
index 631b8ccfc..9ef00ab3f 100644
--- a/src/cpu/simple_thread.hh
+++ b/src/cpu/simple_thread.hh
@@ -424,9 +424,9 @@ class SimpleThread : public ThreadState
     void setStCondFailures(unsigned sc_failures)
     { storeCondFailures = sc_failures; }
 
-    void syscall(int64_t callnum)
+    void syscall(int64_t callnum, Fault *fault)
     {
-        process->syscall(callnum, tc);
+        process->syscall(callnum, tc, fault);
     }
 
     uint64_t readIntRegFlat(int idx) { return intRegs[idx]; }
diff --git a/src/cpu/thread_context.hh b/src/cpu/thread_context.hh
index f966c0aa1..ecbd1a41e 100644
--- a/src/cpu/thread_context.hh
+++ b/src/cpu/thread_context.hh
@@ -264,7 +264,7 @@ class ThreadContext
     // Same with st cond failures.
     virtual Counter readFuncExeInst() = 0;
 
-    virtual void syscall(int64_t callnum) = 0;
+    virtual void syscall(int64_t callnum, Fault *fault) = 0;
 
     // This function exits the thread context in the CPU and returns
     // 1 if the CPU has no more active threads (meaning it's OK to exit);
@@ -471,8 +471,8 @@ class ProxyThreadContext : public ThreadContext
     void setStCondFailures(unsigned sc_failures)
     { actualTC->setStCondFailures(sc_failures); }
 
-    void syscall(int64_t callnum)
-    { actualTC->syscall(callnum); }
+    void syscall(int64_t callnum, Fault *fault)
+    { actualTC->syscall(callnum, fault); }
 
     Counter readFuncExeInst() { return actualTC->readFuncExeInst(); }
 
diff --git a/src/sim/faults.cc b/src/sim/faults.cc
index 93e766526..b0fa6fedb 100644
--- a/src/sim/faults.cc
+++ b/src/sim/faults.cc
@@ -59,6 +59,11 @@ void ReExec::invoke(ThreadContext *tc, const StaticInstPtr &inst)
     tc->pcState(tc->pcState());
 }
 
+void SyscallRetryFault::invoke(ThreadContext *tc, const StaticInstPtr &inst)
+{
+    tc->pcState(tc->pcState());
+}
+
 void GenericPageTableFault::invoke(ThreadContext *tc, const StaticInstPtr &inst)
 {
     bool handled = false;
diff --git a/src/sim/faults.hh b/src/sim/faults.hh
index da0ea22fa..be7aab582 100644
--- a/src/sim/faults.hh
+++ b/src/sim/faults.hh
@@ -72,6 +72,22 @@ class ReExec : public FaultBase
                 StaticInst::nullStaticInstPtr);
 };
 
+/*
+ * This class is needed to allow system call retries to occur for blocking
+ * system calls in SE mode. A retry fault will be generated by the system call
+ * emulation code if blocking conditions arise; the fault is passed up the
+ * function call chain into the CPU model where it is handled by retrying the
+ * syscall instruction on a later tick.
+ */
+class SyscallRetryFault : public FaultBase
+{
+  public:
+    virtual FaultName name() const { return "System call retry fault"; }
+    SyscallRetryFault() {}
+    void invoke(ThreadContext *tc, const StaticInstPtr &inst =
+                StaticInst::nullStaticInstPtr);
+};
+
 class GenericPageTableFault : public FaultBase
 {
   private:
diff --git a/src/sim/process.cc b/src/sim/process.cc
index 4c44f4086..5acfa88b3 100644
--- a/src/sim/process.cc
+++ b/src/sim/process.cc
@@ -269,7 +269,7 @@ Process::map(Addr vaddr, Addr paddr, int size, bool cacheable)
 }
 
 void
-Process::syscall(int64_t callnum, ThreadContext *tc)
+Process::syscall(int64_t callnum, ThreadContext *tc, Fault *fault)
 {
     num_syscalls++;
 
@@ -277,7 +277,7 @@ Process::syscall(int64_t callnum, ThreadContext *tc)
     if (desc == NULL)
         fatal("Syscall %d out of range", callnum);
 
-    desc->doSyscall(callnum, this, tc);
+    desc->doSyscall(callnum, this, tc, fault);
 }
 
 IntReg
diff --git a/src/sim/process.hh b/src/sim/process.hh
index 2191c3cd0..101e4a31c 100644
--- a/src/sim/process.hh
+++ b/src/sim/process.hh
@@ -80,7 +80,7 @@ class Process : public SimObject
     void initState() override;
     DrainState drain() override;
 
-    void syscall(int64_t callnum, ThreadContext *tc);
+    void syscall(int64_t callnum, ThreadContext *tc, Fault *fault);
     virtual TheISA::IntReg getSyscallArg(ThreadContext *tc, int &i) = 0;
     virtual TheISA::IntReg getSyscallArg(ThreadContext *tc, int &i, int width);
     virtual void setSyscallArg(ThreadContext *tc, int i,
diff --git a/src/sim/syscall_desc.cc b/src/sim/syscall_desc.cc
index bb0e80f5f..13b519081 100644
--- a/src/sim/syscall_desc.cc
+++ b/src/sim/syscall_desc.cc
@@ -33,16 +33,21 @@
 
 #include "sim/syscall_desc.hh"
 
+#include <memory>
+
 #include "base/trace.hh"
+#include "base/types.hh"
 #include "config/the_isa.hh"
 #include "cpu/base.hh"
 #include "cpu/thread_context.hh"
+#include "sim/faults.hh"
 #include "sim/process.hh"
 #include "sim/syscall_debug_macros.hh"
 #include "sim/syscall_return.hh"
 
 void
-SyscallDesc::doSyscall(int callnum, Process *process, ThreadContext *tc)
+SyscallDesc::doSyscall(int callnum, Process *process, ThreadContext *tc,
+                       Fault *fault)
 {
     TheISA::IntReg arg[6] M5_VAR_USED;
 
@@ -71,9 +76,10 @@ SyscallDesc::doSyscall(int callnum, Process *process, ThreadContext *tc)
      * blocking behavior, warn that the system call will retry;
      * alternatively, print the return value.
      */
-    if (retval.needsRetry())
+    if (retval.needsRetry()) {
+        *fault = std::make_shared<SyscallRetryFault>();
         DPRINTF_SYSCALL(Base, "%s needs retry\n", _name);
-    else
+    } else
         DPRINTF_SYSCALL(Base, "%s returns %d\n", _name, retval.encodedValue());
 
     if (!(_flags & SyscallDesc::SuppressReturnValue) && !retval.needsRetry())
diff --git a/src/sim/syscall_desc.hh b/src/sim/syscall_desc.hh
index 37aaad14d..b4d24e672 100644
--- a/src/sim/syscall_desc.hh
+++ b/src/sim/syscall_desc.hh
@@ -48,6 +48,8 @@
 
 #include <string>
 
+#include "base/types.hh"
+
 class Process;
 class SyscallReturn;
 class ThreadContext;
@@ -91,7 +93,8 @@ class SyscallDesc {
      * @param proc Handle for the owning Process to pass information
      * @param tc Handle for owning ThreadContext to pass information
      */
-    void doSyscall(int callnum, Process *proc, ThreadContext *tc);
+    void doSyscall(int callnum, Process *proc, ThreadContext *tc,
+                   Fault *fault);
 
     /**
      * Return false if WarnOnce is set and a warning has already been issued.