Lock reworking.

This is a significant change to the use of locks in MuPDF.

Previously, the user had the option of passing us lock/unlock
functions for a single mutex as part of the allocation struct.
Now we remove these entries from the allocation struct, and
make a separate 'locks' struct. This enables people to use
fz_alloc_default with locking.

If multithreaded operation is required, then the user is
required to create FZ_LOCK_MAX mutexes, which will be locked
or unlocked by MuPDF calling the lock/unlock functions within
the new fz_locks_context structure passed in at context creation.

These mutexes are not required to be recursive (they may be, but
MuPDF should never call them in this way). MuPDF avoids deadlocks
by imposing a locking ordering on itself; a thread will never take
lock n, if it already holds any lock i for which 0 <= i <= n.

Currently, there are 4 locks used within MuPDF.

Lock 0: The alloc lock; taken around all calls to user supplied
(or default) allocation functions. Also taken around all accesses
to the refs field of storable items.

Lock 1: The store lock; taken whenever the store data structures
(specifically the linked list pointers) are accessed.

Lock 2: The file lock; taken whenever a thread is accessing the raw
file. We use the debugging macros to insist that this is held
whenever we do a file based seek or read. We also insist that this
is never held when we resolve an indirect reference, as this can
have the effect of moving the file pointer.

Lock 3: The glyphcache lock; taken whenever a thread calls freetype,
or accesses the glyphcache data structures. This introduces some
complexities w.r.t type3 fonts.

Locking can be hugely problematic, so to ease our minds as to
the correctness of this code, we introduce some debugging macros.
These compile away to nothing unless FITZ_DEBUG_LOCKING is defined.

fz_assert_lock_held(ctx, lock) checks that we hold lock.

fz_assert_lock_not_held(ctx, lock) checks that we do not hold lock.

In addition fz_lock_debug_lock and fz_lock_debug_unlock are used
on every fz_lock/fz_unlock to check the validity of the operation
we are performing - in particular it checks that we do/do not already
hold the lock we are trying to take/drop, and that by taking this
lock we are not violating our defined locking order.

The RESOLVE macro (used throughout the code to check whether we need
to resolve an indirect reference) calls fz_assert_lock_not_held to
ensure that we aren't about to resolve an indirect reference (and
hence move the stream pointer) when the file is locked.

In order to implement the file locking properly, pdf_open_stream
(and friends) now lock the file as a side effect (because they
fz_seek to the start of the stream). The lock is automatically
dropped on an fz_close of such streams.

Previously, the glyph cache was created in a context when it was first
required; this presents problems as it can be shared between several
contexts or not, depending on whether it is created before the
contexts are cloned. We now always create it at startup, so it is
always shared.

This means that we need reference counting for the glyph caches.
Added here.

In fz_render_glyph, we take the glyph cache lock, and check to see
whether the glyph is in the cache. If it is, we bump the refcount,
drop the lock and returned the cached character. If it is not, we
need to render the character.

For freetype based fonts we keep the lock throughout the rendering
process, thus ensuring that freetype is only called in a single
threaded manner.

For type3 fonts, however, we need to invoke the interpreter again
to render the glyph streams. This can require reentrance to this
routine. We therefore drop the glyph cache lock, call the
interpreter to render us our pixmap, and take the lock again.

This dropping and retaking of the lock introduces a possible race
condition; 2 threads may try to render the same character at the
same time. We therefore modify our hash table insert routines to
behave differently if it comes to insert an entry only to find
that an entry with the same key is already there.

We spot this case; if we have just rendered a type3 glyph and when
we try to insert it into the cache discover that someone has beaten
us to it, we just discard our entry and use the cached one.

Hopefully this will seldom be a problem in practise; to solve it
properly would require greater complexity (probably involving
spotting that another thread is already working on the desired
rendering, and sleeping on a semaphore until it completes).

26 files changed, 447 insertions, 118 deletions

diff --git a/apps/mupdfclean.c b/apps/mupdfclean.c
index bd04825a..3130e3f0 100644
--- a/apps/mupdfclean.c
+++ b/apps/mupdfclean.c
@@ -782,7 +782,7 @@ int main(int argc, char **argv)
 	if (argc - fz_optind > 0)
 		subset = 1;
 
-	ctx = fz_new_context(NULL, FZ_STORE_UNLIMITED);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_UNLIMITED);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
diff --git a/apps/mupdfdraw.c b/apps/mupdfdraw.c
index 6325c791..c58b0b2e 100644
--- a/apps/mupdfdraw.c
+++ b/apps/mupdfdraw.c
@@ -408,7 +408,7 @@ int main(int argc, char **argv)
 	if (accelerate)
 		fz_accelerate();
 
-	ctx = fz_new_context(NULL, FZ_STORE_DEFAULT);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_DEFAULT);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
diff --git a/apps/mupdfextract.c b/apps/mupdfextract.c
index 1407f7f3..f309f39a 100644
--- a/apps/mupdfextract.c
+++ b/apps/mupdfextract.c
@@ -182,7 +182,7 @@ int main(int argc, char **argv)
 
 	infile = argv[fz_optind++];
 
-	ctx = fz_new_context(NULL, FZ_STORE_UNLIMITED);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_UNLIMITED);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
diff --git a/apps/mupdfinfo.c b/apps/mupdfinfo.c
index c6c6b35c..3c98c772 100644
--- a/apps/mupdfinfo.c
+++ b/apps/mupdfinfo.c
@@ -971,7 +971,7 @@ int main(int argc, char **argv)
 	if (fz_optind == argc)
 		infousage();
 
-	ctx = fz_new_context(NULL, FZ_STORE_UNLIMITED);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_UNLIMITED);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
diff --git a/apps/mupdfshow.c b/apps/mupdfshow.c
index 53578fd7..8dcdd17d 100644
--- a/apps/mupdfshow.c
+++ b/apps/mupdfshow.c
@@ -198,7 +198,7 @@ int main(int argc, char **argv)
 
 	filename = argv[fz_optind++];
 
-	ctx = fz_new_context(NULL, FZ_STORE_UNLIMITED);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_UNLIMITED);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
diff --git a/apps/muxpsdraw.c b/apps/muxpsdraw.c
index 2bf27d74..6856338c 100644
--- a/apps/muxpsdraw.c
+++ b/apps/muxpsdraw.c
@@ -307,7 +307,7 @@ int main(int argc, char **argv)
 	if (accelerate)
 		fz_accelerate();
 
-	ctx = fz_new_context(NULL, FZ_STORE_DEFAULT);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_DEFAULT);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
diff --git a/apps/win_main.c b/apps/win_main.c
index c421fac7..22948de4 100644
--- a/apps/win_main.c
+++ b/apps/win_main.c
@@ -752,7 +752,7 @@ WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nShow
 
 	fz_accelerate();
 
-	ctx = fz_new_context(NULL, FZ_STORE_DEFAULT);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_DEFAULT);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
diff --git a/apps/x11_main.c b/apps/x11_main.c
index f9b1dee1..f9c00c99 100644
--- a/apps/x11_main.c
+++ b/apps/x11_main.c
@@ -623,7 +623,7 @@ int main(int argc, char **argv)
 	struct timeval tmo;
 	struct timeval *timeout;
 
-	ctx = fz_new_context(NULL, FZ_STORE_DEFAULT);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_DEFAULT);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
diff --git a/draw/draw_glyph.c b/draw/draw_glyph.c
index 49f04fe8..00eaf460 100644
--- a/draw/draw_glyph.c
+++ b/draw/draw_glyph.c
@@ -8,6 +8,7 @@ typedef struct fz_glyph_key_s fz_glyph_key;
 
 struct fz_glyph_cache_s
 {
+	int refs;
 	fz_hash_table *hash;
 	int total;
 };
@@ -37,10 +38,12 @@ fz_new_glyph_cache_context(fz_context *ctx)
 		fz_rethrow(ctx);
 	}
 	cache->total = 0;
+	cache->refs = 1;
 
 	ctx->glyph_cache = cache;
 }
 
+/* The glyph cache lock is always held when this function is called. */
 static void
 fz_evict_glyph_cache(fz_context *ctx)
 {
@@ -70,10 +73,25 @@ fz_free_glyph_cache_context(fz_context *ctx)
 	if (!ctx->glyph_cache)
 		return;
 
-	fz_evict_glyph_cache(ctx);
-	fz_free_hash(ctx, ctx->glyph_cache->hash);
-	fz_free(ctx, ctx->glyph_cache);
-	ctx->glyph_cache = NULL;
+	fz_lock(ctx, FZ_LOCK_GLYPHCACHE);
+	ctx->glyph_cache->refs--;
+	if (ctx->glyph_cache->refs == 0)
+	{
+		fz_evict_glyph_cache(ctx);
+		fz_free_hash(ctx, ctx->glyph_cache->hash);
+		fz_free(ctx, ctx->glyph_cache);
+		ctx->glyph_cache = NULL;
+	}
+	fz_unlock(ctx, FZ_LOCK_GLYPHCACHE);
+}
+
+fz_glyph_cache *
+fz_glyph_cache_keep(fz_context *ctx)
+{
+	fz_lock(ctx, FZ_LOCK_GLYPHCACHE);
+	ctx->glyph_cache->refs++;
+	fz_unlock(ctx, FZ_LOCK_GLYPHCACHE);
+	return ctx->glyph_cache;
 }
 
 fz_pixmap *
@@ -92,8 +110,6 @@ fz_render_glyph(fz_context *ctx, fz_font *font, int gid, fz_matrix ctm, fz_color
 	fz_pixmap *val;
 	float size = fz_matrix_expansion(ctm);
 
-	if (!ctx->glyph_cache)
-		fz_new_glyph_cache_context(ctx);
 	cache = ctx->glyph_cache;
 
 	if (size > MAX_FONT_SIZE)
@@ -113,25 +129,49 @@ fz_render_glyph(fz_context *ctx, fz_font *font, int gid, fz_matrix ctm, fz_color
 	key.e = (ctm.e - floorf(ctm.e)) * 256;
 	key.f = (ctm.f - floorf(ctm.f)) * 256;
 
+	fz_lock(ctx, FZ_LOCK_GLYPHCACHE);
 	val = fz_hash_find(ctx, cache->hash, &key);
 	if (val)
-		return fz_keep_pixmap(ctx, val);
+	{
+		fz_keep_pixmap(ctx, val);
+		fz_unlock(ctx, FZ_LOCK_GLYPHCACHE);
+		return val;
+	}
 
 	ctm.e = floorf(ctm.e) + key.e / 256.0f;
 	ctm.f = floorf(ctm.f) + key.f / 256.0f;
 
-	if (font->ft_face)
-	{
-		val = fz_render_ft_glyph(ctx, font, gid, ctm);
-	}
-	else if (font->t3procs)
+	fz_try(ctx)
 	{
-		val = fz_render_t3_glyph(ctx, font, gid, ctm, model);
+		if (font->ft_face)
+		{
+			val = fz_render_ft_glyph(ctx, font, gid, ctm);
+		}
+		else if (font->t3procs)
+		{
+			/* We drop the glyphcache here, and execute the t3
+			 * glyph code. The danger here is that some other
+			 * thread will come along, and want the same glyph
+			 * too. If it does, we may both end up rendering
+			 * pixmaps. We cope with this later on, by ensuring
+			 * that only one gets inserted into the cache. If
+			 * we insert ours to find one already there, we
+			 * abandon ours, and use the one there already.
+			 */
+			fz_unlock(ctx, FZ_LOCK_GLYPHCACHE);
+			val = fz_render_t3_glyph(ctx, font, gid, ctm, model);
+			fz_lock(ctx, FZ_LOCK_GLYPHCACHE);
+		}
+		else
+		{
+			fz_warn(ctx, "assert: uninitialized font structure");
+			val = NULL;
+		}
 	}
-	else
+	fz_catch(ctx)
 	{
-		fz_warn(ctx, "assert: uninitialized font structure");
-		return NULL;
+		fz_unlock(ctx, FZ_LOCK_GLYPHCACHE);
+		fz_rethrow(ctx);
 	}
 
 	if (val)
@@ -142,8 +182,14 @@ fz_render_glyph(fz_context *ctx, fz_font *font, int gid, fz_matrix ctm, fz_color
 				fz_evict_glyph_cache(ctx);
 			fz_try(ctx)
 			{
-				fz_hash_insert(ctx, cache->hash, &key, val);
-				fz_keep_font(ctx, key.font);
+				fz_pixmap *pix = fz_hash_insert(ctx, cache->hash, &key, val);
+				if (pix)
+				{
+					fz_drop_pixmap(ctx, val);
+					val = pix;
+				}
+				else
+					fz_keep_font(ctx, key.font);
 				val = fz_keep_pixmap(ctx, val);
 			}
 			fz_catch(ctx)
@@ -151,10 +197,9 @@ fz_render_glyph(fz_context *ctx, fz_font *font, int gid, fz_matrix ctm, fz_color
 				fz_warn(ctx, "Failed to encache glyph - continuing");
 			}
 			cache->total += val->w * val->h;
-			return val;
 		}
-		return val;
 	}
 
-	return NULL;
+	fz_unlock(ctx, FZ_LOCK_GLYPHCACHE);
+	return val;
 }
diff --git a/fitz/base_context.c b/fitz/base_context.c
index b0ffe2a4..47d4ba79 100644
--- a/fitz/base_context.c
+++ b/fitz/base_context.c
@@ -40,7 +40,7 @@ fz_free_context(fz_context *ctx)
  * that aren't shared between contexts.
  */
 static fz_context *
-new_context_phase1(fz_alloc_context *alloc)
+new_context_phase1(fz_alloc_context *alloc, fz_locks_context *locks)
 {
 	fz_context *ctx;
 
@@ -49,6 +49,7 @@ new_context_phase1(fz_alloc_context *alloc)
 		return NULL;
 	memset(ctx, 0, sizeof *ctx);
 	ctx->alloc = alloc;
+	ctx->locks = locks;
 
 	ctx->glyph_cache = NULL;
 
@@ -84,19 +85,23 @@ cleanup:
 }
 
 fz_context *
-fz_new_context(fz_alloc_context *alloc, unsigned int max_store)
+fz_new_context(fz_alloc_context *alloc, fz_locks_context *locks, unsigned int max_store)
 {
 	fz_context *ctx;
 
 	if (!alloc)
 		alloc = &fz_alloc_default;
 
-	ctx = new_context_phase1(alloc);
+	if (!locks)
+		locks = &fz_locks_default;
+
+	ctx = new_context_phase1(alloc, locks);
 
 	/* Now initialise sections that are shared */
 	fz_try(ctx)
 	{
 		fz_new_store_context(ctx, max_store);
+		fz_new_glyph_cache_context(ctx);
 	}
 	fz_catch(ctx)
 	{
@@ -114,10 +119,11 @@ fz_clone_context(fz_context *ctx)
 
 	/* We cannot safely clone the context without having locking/
 	 * unlocking functions. */
-	if (ctx == NULL || ctx->alloc == NULL || ctx->alloc->lock == NULL || ctx->alloc->unlock == NULL)
+	if (ctx == NULL || ctx->alloc == NULL || ctx->locks == &fz_locks_default)
 		return NULL;
 
-	new_ctx = new_context_phase1(ctx->alloc);
+	new_ctx = new_context_phase1(ctx->alloc, ctx->locks);
 	new_ctx->store = fz_store_keep(ctx);
+	new_ctx->glyph_cache = fz_glyph_cache_keep(ctx);
 	return new_ctx;
 }
diff --git a/fitz/base_hash.c b/fitz/base_hash.c
index b0a3b3f4..630f6b6a 100644
--- a/fitz/base_hash.c
+++ b/fitz/base_hash.c
@@ -147,7 +147,7 @@ fz_hash_find(fz_context *ctx, fz_hash_table *table, void *key)
 	}
 }
 
-void
+void *
 fz_hash_insert(fz_context *ctx, fz_hash_table *table, void *key, void *val)
 {
 	fz_hash_entry *ents;
@@ -170,11 +170,14 @@ fz_hash_insert(fz_context *ctx, fz_hash_table *table, void *key, void *val)
 			memcpy(ents[pos].key, key, table->keylen);
 			ents[pos].val = val;
 			table->load ++;
-			return;
+			return NULL;
 		}
 
 		if (memcmp(key, ents[pos].key, table->keylen) == 0)
+		{
 			fz_warn(ctx, "assert: overwrite hash slot");
+			return ents[pos].val;
+		}
 
 		pos = (pos + 1) % size;
 	}
diff --git a/fitz/base_memory.c b/fitz/base_memory.c
index bae83cbd..a43c06a0 100644
--- a/fitz/base_memory.c
+++ b/fitz/base_memory.c
@@ -6,16 +6,19 @@ do_scavenging_malloc(fz_context *ctx, unsigned int size)
 	void *p;
 	int phase = 0;
 
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_STORE);
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	do {
 		p = ctx->alloc->malloc(ctx->alloc->user, size);
 		if (p != NULL)
 		{
-			fz_unlock(ctx);
+			fz_unlock(ctx, FZ_LOCK_ALLOC);
+			fz_unlock(ctx, FZ_LOCK_STORE);
 			return p;
 		}
 	} while (fz_store_scavenge(ctx, size, &phase));
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
+	fz_unlock(ctx, FZ_LOCK_STORE);
 
 	return NULL;
 }
@@ -26,16 +29,19 @@ do_scavenging_realloc(fz_context *ctx, void *p, unsigned int size)
 	void *q;
 	int phase = 0;
 
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_STORE);
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	do {
 		q = ctx->alloc->realloc(ctx->alloc->user, p, size);
 		if (q != NULL)
 		{
-			fz_unlock(ctx);
+			fz_unlock(ctx, FZ_LOCK_ALLOC);
+			fz_unlock(ctx, FZ_LOCK_STORE);
 			return q;
 		}
 	} while (fz_store_scavenge(ctx, size, &phase));
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
+	fz_unlock(ctx, FZ_LOCK_STORE);
 
 	return NULL;
 }
@@ -177,9 +183,9 @@ fz_resize_array_no_throw(fz_context *ctx, void *p, unsigned int count, unsigned
 void
 fz_free(fz_context *ctx, void *p)
 {
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	ctx->alloc->free(ctx->alloc->user, p);
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
 }
 
 char *
@@ -226,3 +232,105 @@ fz_alloc_context fz_alloc_default =
 	fz_realloc_default,
 	fz_free_default
 };
+
+static void
+fz_lock_default(void *user, int lock)
+{
+}
+
+static void
+fz_unlock_default(void *user, int lock)
+{
+}
+
+fz_locks_context fz_locks_default =
+{
+	NULL,
+	fz_lock_default,
+	fz_unlock_default
+};
+
+#ifdef FITZ_DEBUG_LOCKING
+
+enum
+{
+	FZ_LOCK_DEBUG_CONTEXT_MAX = 100
+};
+
+fz_context *fz_lock_debug_contexts[FZ_LOCK_DEBUG_CONTEXT_MAX];
+int fz_locks_debug[FZ_LOCK_DEBUG_CONTEXT_MAX][FZ_LOCK_MAX];
+
+static int find_context(fz_context *ctx)
+{
+	int i;
+
+	for (i = 0; i < FZ_LOCK_DEBUG_CONTEXT_MAX; i++)
+	{
+		if (fz_lock_debug_contexts[i] == ctx)
+			return i;
+		if (fz_lock_debug_contexts[i] == NULL)
+		{
+			fz_lock_debug_contexts[i] = ctx;
+			return i;
+		}
+	}
+	return -1;
+}
+
+void
+fz_assert_lock_held(fz_context *ctx, int lock)
+{
+	int idx = find_context(ctx);
+	if (idx < 0)
+		return;
+
+	if (fz_locks_debug[idx][lock] == 0)
+		fprintf(stderr, "Lock %d not held when expected\n", lock);
+}
+
+void
+fz_assert_lock_not_held(fz_context *ctx, int lock)
+{
+	int idx = find_context(ctx);
+	if (idx < 0)
+		return;
+
+	if (fz_locks_debug[idx][lock] != 0)
+		fprintf(stderr, "Lock %d held when not expected\n", lock);
+}
+
+void fz_lock_debug_lock(fz_context *ctx, int lock)
+{
+	int i;
+	int idx = find_context(ctx);
+	if (idx < 0)
+		return;
+
+	if (fz_locks_debug[idx][lock] != 0)
+	{
+		fprintf(stderr, "Attempt to take lock %d when held already!\n", lock);
+	}
+	for (i = lock-1; i >= 0; i--)
+	{
+		if (fz_locks_debug[idx][i] != 0)
+		{
+			fprintf(stderr, "Lock ordering violation: Attempt to take lock %d when %d held already!\n", lock, i);
+		}
+	}
+	fz_locks_debug[idx][lock] = 1;
+}
+
+void fz_lock_debug_unlock(fz_context *ctx, int lock)
+{
+	int idx = find_context(ctx);
+	if (idx < 0)
+		return;
+
+	if (fz_locks_debug[idx][lock] == 0)
+	{
+		fprintf(stderr, "Attempt to release lock %d when not held!\n", lock);
+	}
+	fz_locks_debug[idx][lock] = 0;
+}
+
+#endif
diff --git a/fitz/base_object.c b/fitz/base_object.c
index a3b38c49..964c5bb4 100644
--- a/fitz/base_object.c
+++ b/fitz/base_object.c
@@ -155,8 +155,12 @@ int fz_is_indirect(fz_obj *obj)
 }
 
 #define RESOLVE(obj) \
-	do { if (obj && obj->kind == FZ_INDIRECT) \
-		obj = fz_resolve_indirect(obj); \
+	do { \
+		if (obj && obj->kind == FZ_INDIRECT) \
+		{\
+			fz_assert_lock_not_held(obj->ctx, FZ_LOCK_FILE); \
+			obj = fz_resolve_indirect(obj); \
+		} \
 	} while (0)
 
 int fz_is_null(fz_obj *obj)
diff --git a/fitz/fitz.h b/fitz/fitz.h
index 9ce721ea..335088e5 100644
--- a/fitz/fitz.h
+++ b/fitz/fitz.h
@@ -110,6 +110,7 @@ typedef struct fz_error_context_s fz_error_context;
 typedef struct fz_warn_context_s fz_warn_context;
 typedef struct fz_font_context_s fz_font_context;
 typedef struct fz_aa_context_s fz_aa_context;
+typedef struct fz_locks_context_s fz_locks_context;
 typedef struct fz_store_s fz_store;
 typedef struct fz_glyph_cache_s fz_glyph_cache;
 typedef struct fz_context_s fz_context;
@@ -120,8 +121,6 @@ struct fz_alloc_context_s
 	void *(*malloc)(void *, unsigned int);
 	void *(*realloc)(void *, void *, unsigned int);
 	void (*free)(void *, void *);
-	void (*lock)(void *);
-	void (*unlock)(void *);
 };
 
 /* Default allocator */
@@ -363,6 +362,7 @@ void fz_flush_warnings(fz_context *ctx);
 struct fz_context_s
 {
 	fz_alloc_context *alloc;
+	fz_locks_context *locks;
 	fz_error_context *error;
 	fz_warn_context *warn;
 	fz_font_context *font;
@@ -371,27 +371,85 @@ struct fz_context_s
 	fz_glyph_cache *glyph_cache;
 };
 
-fz_context *fz_new_context(fz_alloc_context *alloc, unsigned int max_store);
+fz_context *fz_new_context(fz_alloc_context *alloc, fz_locks_context *locks, unsigned int max_store);
 fz_context *fz_clone_context(fz_context *ctx);
 void fz_free_context(fz_context *ctx);
 
 void fz_new_aa_context(fz_context *ctx);
 void fz_free_aa_context(fz_context *ctx);
 
+/* Locking functions
+ *
+ * MuPDF is kept deliberately free of any knowledge of particular threading
+ * systems. As such, in order for safe multi-threaded operation, we rely on
+ * callbacks to client provided functions.
+ *
+ * A client is expected to provide FZ_LOCK_MAX mutexes, and a function to
+ * lock/unlock each of them. These may be recursive mutexes, but do not have
+ * to be.
+ *
+ * If a client does not intend to use multiple threads, then it may pass
+ * NULL instead of the address of a lock structure.
+ *
+ * In order to avoid deadlocks, we have 1 simple rules internally as to how
+ * we use locks: We can never take lock n when we already hold any lock i,
+ * where 0 <= i <= n. In order to verify this, we have some debugging code
+ * built in, that is enabled by defining FITZ_DEBUG_LOCKING.
+ */
+
+#if defined(MEMENTO) || defined(DEBUG)
+#define FITZ_DEBUG_LOCKING
+#endif
+
+struct fz_locks_context_s
+{
+	void *user;
+	void (*lock)(void *, int);
+	void (*unlock)(void *, int);
+};
+
+enum {
+	FZ_LOCK_ALLOC = 0,
+	FZ_LOCK_STORE,
+	FZ_LOCK_FILE,
+	FZ_LOCK_GLYPHCACHE,
+	FZ_LOCK_MAX
+};
+
+/* Default locks */
+extern fz_locks_context fz_locks_default;
+
+#ifdef FITZ_DEBUG_LOCKING
+
+void fz_assert_lock_held(fz_context *ctx, int lock);
+void fz_assert_lock_not_held(fz_context *ctx, int lock);
+void fz_lock_debug_lock(fz_context *ctx, int lock);
+void fz_lock_debug_unlock(fz_context *ctx, int lock);
+
+#else
+
+#define fz_assert_lock_held(A,B) do { } while (0)
+#define fz_assert_lock_not_held(A,B) do { } while (0)
+#define fz_lock_debug_lock(A,B) do { } while (0)
+#define fz_lock_debug_unlock(A,B) do { } while (0)
+
+#endif /* !FITZ_DEBUG_LOCKING */
+
 static inline void
-fz_lock(fz_context *ctx)
+fz_lock(fz_context *ctx, int lock)
 {
-	if (ctx->alloc->lock)
-		ctx->alloc->lock(ctx->alloc->user);
+	fz_lock_debug_lock(ctx, lock);
+	ctx->locks->lock(ctx->locks->user, lock);
 }
 
 static inline void
-fz_unlock(fz_context *ctx)
+fz_unlock(fz_context *ctx, int lock)
 {
-	if (ctx->alloc->unlock)
-		ctx->alloc->unlock(ctx->alloc->user);
+	fz_lock_debug_unlock(ctx, lock);
+	ctx->locks->unlock(ctx->locks->user, lock);
 }
 
+
 /*
  * Basic runtime and utility functions
  */
@@ -451,7 +509,7 @@ void fz_empty_hash(fz_context *ctx, fz_hash_table *table);
 void fz_free_hash(fz_context *ctx, fz_hash_table *table);
 
 void *fz_hash_find(fz_context *ctx, fz_hash_table *table, void *key);
-void fz_hash_insert(fz_context *ctx, fz_hash_table *table, void *key, void *val);
+void *fz_hash_insert(fz_context *ctx, fz_hash_table *table, void *key, void *val);
 void fz_hash_remove(fz_context *ctx, fz_hash_table *table, void *key);
 
 int fz_hash_len(fz_context *ctx, fz_hash_table *table);
@@ -781,6 +839,7 @@ struct fz_stream_s
 	int pos;
 	int avail;
 	int bits;
+	int locked;
 	unsigned char *bp, *rp, *wp, *ep;
 	void *state;
 	int (*read)(fz_stream *stm, unsigned char *buf, int len);
@@ -795,6 +854,7 @@ fz_stream *fz_open_file_w(fz_context *ctx, const wchar_t *filename); /* only on
 fz_stream *fz_open_buffer(fz_context *ctx, fz_buffer *buf);
 fz_stream *fz_open_memory(fz_context *ctx, unsigned char *data, int len);
 void fz_close(fz_stream *stm);
+void fz_lock_stream(fz_stream *stm);
 
 fz_stream *fz_new_stream(fz_context *ctx, void*, int(*)(fz_stream*, unsigned char*, int), void(*)(fz_context *, void *));
 fz_stream *fz_keep_stream(fz_stream *stm);
@@ -1188,6 +1248,7 @@ void fz_debug_path(fz_context *ctx, fz_path *, int indent);
  */
 
 void fz_new_glyph_cache_context(fz_context *ctx);
+fz_glyph_cache *fz_glyph_cache_keep(fz_context *ctx);
 void fz_free_glyph_cache_context(fz_context *ctx);
 
 fz_pixmap *fz_render_ft_glyph(fz_context *ctx, fz_font *font, int cid, fz_matrix trm);
diff --git a/fitz/res_font.c b/fitz/res_font.c
index f8dcfb90..298e637e 100644
--- a/fitz/res_font.c
+++ b/fitz/res_font.c
@@ -345,6 +345,7 @@ fz_copy_ft_bitmap(fz_context *ctx, int left, int top, FT_Bitmap *bitmap)
 	return pixmap;
 }
 
+/* The glyph cache lock is always taken when this is called. */
 fz_pixmap *
 fz_render_ft_glyph(fz_context *ctx, fz_font *font, int gid, fz_matrix trm)
 {
diff --git a/fitz/res_store.c b/fitz/res_store.c
index 398e8ece..c29a74bc 100644
--- a/fitz/res_store.c
+++ b/fitz/res_store.c
@@ -63,10 +63,10 @@ fz_keep_storable(fz_context *ctx, fz_storable *s)
 {
 	if (s == NULL)
 		return NULL;
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	if (s->refs > 0)
 		s->refs++;
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
 	return s;
 }
 
@@ -77,7 +77,7 @@ fz_drop_storable(fz_context *ctx, fz_storable *s)
 
 	if (s == NULL)
 		return;
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	if (s->refs < 0)
 	{
 		/* It's a static object. Dropping does nothing. */
@@ -91,7 +91,7 @@ fz_drop_storable(fz_context *ctx, fz_storable *s)
 		 * itself without any operations on the fz_store. */
 		do_free = 1;
 	}
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
 	if (do_free)
 		s->free(ctx, s);
 }
@@ -100,6 +100,7 @@ static void
 evict(fz_context *ctx, fz_item *item)
 {
 	fz_store *store = ctx->store;
+	int drop;
 
 	store->size -= item->size;
 	/* Unlink from the linked list */
@@ -121,7 +122,10 @@ evict(fz_context *ctx, fz_item *item)
 		fz_hash_remove(ctx, store->hash, &refkey);
 	}
 	/* Drop a reference to the value (freeing if required) */
-	if (item->val->refs > 0 && --item->val->refs == 0)
+	fz_lock(ctx, FZ_LOCK_ALLOC);
+	drop = (item->val->refs > 0 && --item->val->refs == 0);
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
+	if (drop)
 		item->val->free(ctx, item->val);
 	/* Always drops the key and free the item */
 	fz_drop_obj(item->key);
@@ -135,6 +139,7 @@ ensure_space(fz_context *ctx, unsigned int tofree)
 	unsigned int count;
 	fz_store *store = ctx->store;
 
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	/* First check that we *can* free tofree; if not, we'd rather not
 	 * cache this. */
 	count = 0;
@@ -150,7 +155,10 @@ ensure_space(fz_context *ctx, unsigned int tofree)
 
 	/* If we ran out of items to search, then we can never free enough */
 	if (item == NULL)
+	{
+		fz_unlock(ctx, FZ_LOCK_ALLOC);
 		return 1;
+	}
 
 	/* Actually free the items */
 	count = 0;
@@ -161,12 +169,15 @@ ensure_space(fz_context *ctx, unsigned int tofree)
 		{
 			/* Free this item */
 			count += item->size;
+			fz_unlock(ctx, FZ_LOCK_ALLOC);
 			evict(ctx, item);
 
 			if (count >= tofree)
-				break;
+				return 0;
+			fz_lock(ctx, FZ_LOCK_ALLOC);
 		}
 	}
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
 
 	return 0;
 }
@@ -196,11 +207,11 @@ fz_store_item(fz_context *ctx, fz_obj *key, void *val_, unsigned int itemsize)
 		return;
 	}
 
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_STORE);
 	size = store->size + itemsize;
 	if (store->max != FZ_STORE_UNLIMITED && size > store->max && ensure_space(ctx, size - store->max))
 	{
-		fz_unlock(ctx);
+		fz_unlock(ctx, FZ_LOCK_STORE);
 		fz_free(ctx, item);
 		return;
 	}
@@ -224,14 +235,16 @@ fz_store_item(fz_context *ctx, fz_obj *key, void *val_, unsigned int itemsize)
 		}
 		fz_catch(ctx)
 		{
-			fz_unlock(ctx);
+			fz_unlock(ctx, FZ_LOCK_STORE);
 			fz_free(ctx, item);
 			return;
 		}
 	}
 	/* Now we can never fail, bump the ref */
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	if (val->refs > 0)
 		val->refs++;
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
 	/* Regardless of whether it's indexed, it goes into the linked list */
 	item->next = store->head;
 	if (item->next)
@@ -240,7 +253,7 @@ fz_store_item(fz_context *ctx, fz_obj *key, void *val_, unsigned int itemsize)
 		store->tail = item;
 	store->head = item;
 	item->prev = NULL;
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_STORE);
 }
 
 void *
@@ -256,7 +269,7 @@ fz_find_item(fz_context *ctx, fz_store_free_fn *free, fz_obj *key)
 	if (!key)
 		return NULL;
 
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_STORE);
 	if (fz_is_indirect(key))
 	{
 		/* We can find objects keyed on indirected objects quickly */
@@ -295,12 +308,14 @@ fz_find_item(fz_context *ctx, fz_store_free_fn *free, fz_obj *key)
 		item->prev = NULL;
 		store->head = item;
 		/* And bump the refcount before returning */
+		fz_lock(ctx, FZ_LOCK_ALLOC);
 		if (item->val->refs > 0)
 			item->val->refs++;
-		fz_unlock(ctx);
+		fz_unlock(ctx, FZ_LOCK_ALLOC);
+		fz_unlock(ctx, FZ_LOCK_STORE);
 		return (void *)item->val;
 	}
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_STORE);
 
 	return NULL;
 }
@@ -311,8 +326,9 @@ fz_remove_item(fz_context *ctx, fz_store_free_fn *free, fz_obj *key)
 	struct refkey refkey;
 	fz_item *item;
 	fz_store *store = ctx->store;
+	int drop;
 
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_STORE);
 	if (fz_is_indirect(key))
 	{
 		/* We can find objects keyed on indirect objects quickly */
@@ -340,12 +356,15 @@ fz_remove_item(fz_context *ctx, fz_store_free_fn *free, fz_obj *key)
 			item->prev->next = item->next;
 		else
 			store->head = item->next;
-		if (item->val->refs > 0 && --item->val->refs == 0)
+		fz_lock(ctx, FZ_LOCK_ALLOC);
+		drop = (item->val->refs > 0 && --item->val->refs == 0);
+		fz_unlock(ctx, FZ_LOCK_ALLOC);
+		if (drop)
 			item->val->free(ctx, item->val);
 		fz_drop_obj(item->key);
 		fz_free(ctx, item);
 	}
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_STORE);
 }
 
 void
@@ -357,14 +376,14 @@ fz_empty_store(fz_context *ctx)
 	if (store == NULL)
 		return;
 
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_STORE);
 	/* Run through all the items in the store */
 	for (item = store->head; item; item = next)
 	{
 		next = item->next;
 		evict(ctx, item);
 	}
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_STORE);
 }
 
 fz_store *
@@ -372,9 +391,9 @@ fz_store_keep(fz_context *ctx)
 {
 	if (ctx == NULL || ctx->store == NULL)
 		return NULL;
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	ctx->store->refs++;
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
 	return ctx->store;
 }
 
@@ -384,9 +403,9 @@ fz_free_store_context(fz_context *ctx)
 	int refs;
 	if (ctx == NULL || ctx->store == NULL)
 		return;
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_ALLOC);
 	refs = --ctx->store->refs;
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_ALLOC);
 	if (refs != 0)
 		return;
 
@@ -404,10 +423,12 @@ fz_debug_store(fz_context *ctx)
 
 	printf("-- resource store contents --\n");
 
-	fz_lock(ctx);
+	fz_lock(ctx, FZ_LOCK_STORE);
 	for (item = store->head; item; item = item->next)
 	{
+		fz_lock(ctx, FZ_LOCK_ALLOC);
 		printf("store[*][refs=%d][size=%d] ", item->val->refs, item->size);
+		fz_unlock(ctx, FZ_LOCK_ALLOC);
 		if (fz_is_indirect(item->key))
 		{
 			printf("(%d %d R) ", fz_to_num(item->key), fz_to_gen(item->key));
@@ -415,7 +436,7 @@ fz_debug_store(fz_context *ctx)
 			fz_debug_obj(item->key);
 		printf(" = %p\n", item->val);
 	}
-	fz_unlock(ctx);
+	fz_unlock(ctx, FZ_LOCK_STORE);
 }
 
 static int
diff --git a/fitz/stm_open.c b/fitz/stm_open.c
index 6a987103..e634f729 100644
--- a/fitz/stm_open.c
+++ b/fitz/stm_open.c
@@ -24,6 +24,7 @@ fz_new_stream(fz_context *ctx, void *state,
 
 	stm->bits = 0;
 	stm->avail = 0;
+	stm->locked = 0;
 
 	stm->bp = stm->buf;
 	stm->rp = stm->bp;
@@ -39,6 +40,16 @@ fz_new_stream(fz_context *ctx, void *state,
 	return stm;
 }
 
+void
+fz_lock_stream(fz_stream *stm)
+{
+	if (stm)
+	{
+		fz_lock(stm->ctx, FZ_LOCK_FILE);
+		stm->locked = 1;
+	}
+}
+
 fz_stream *
 fz_keep_stream(fz_stream *stm)
 {
@@ -56,6 +67,8 @@ fz_close(fz_stream *stm)
 	{
 		if (stm->close)
 			stm->close(stm->ctx, stm->state);
+		if (stm->locked)
+			fz_unlock(stm->ctx, FZ_LOCK_FILE);
 		fz_free(stm->ctx, stm);
 	}
 }
@@ -65,6 +78,7 @@ fz_close(fz_stream *stm)
 static int read_file(fz_stream *stm, unsigned char *buf, int len)
 {
 	int n = read(*(int*)stm->state, buf, len);
+	fz_assert_lock_held(stm->ctx, FZ_LOCK_FILE);
 	if (n < 0)
 		fz_throw(stm->ctx, "read error: %s", strerror(errno));
 	return n;
@@ -73,6 +87,7 @@ static int read_file(fz_stream *stm, unsigned char *buf, int len)
 static void seek_file(fz_stream *stm, int offset, int whence)
 {
 	int n = lseek(*(int*)stm->state, offset, whence);
+	fz_assert_lock_held(stm->ctx, FZ_LOCK_FILE);
 	if (n < 0)
 		fz_throw(stm->ctx, "cannot lseek: %s", strerror(errno));
 	stm->pos = n;
diff --git a/pdf/pdf_cmap_load.c b/pdf/pdf_cmap_load.c
index fcf30ad0..3257516c 100644
--- a/pdf/pdf_cmap_load.c
+++ b/pdf/pdf_cmap_load.c
@@ -38,7 +38,6 @@ pdf_load_embedded_cmap(pdf_document *xref, fz_obj *stmobj)
 
 	fz_try(ctx)
 	{
-
 		file = pdf_open_stream(xref, fz_to_num(stmobj), fz_to_gen(stmobj));
 		phase = 1;
 		cmap = pdf_load_cmap(ctx, file);
diff --git a/pdf/pdf_function.c b/pdf/pdf_function.c
index ff602021..23acc4db 100644
--- a/pdf/pdf_function.c
+++ b/pdf/pdf_function.c
@@ -836,8 +836,10 @@ load_postscript_func(pdf_function *func, pdf_document *xref, fz_obj *dict, int n
 	int tok;
 	int len;
 	fz_context *ctx = xref->ctx;
+	int locked = 0;
 
 	fz_var(stream);
+	fz_var(locked);
 
 	fz_try(ctx)
 	{
@@ -856,15 +858,16 @@ load_postscript_func(pdf_function *func, pdf_document *xref, fz_obj *dict, int n
 		codeptr = 0;
 		parse_code(func, stream, &codeptr);
 	}
-	fz_catch(ctx)
+	fz_always(ctx)
 	{
 		fz_close(stream);
+	}
+	fz_catch(ctx)
+	{
 		fz_throw(ctx, "cannot parse calculator function (%d %d R)", num, gen);
 	}
 
 	func->size += func->u.p.cap * sizeof(psobj);
-
-	fz_close(stream);
 }
 
 static void
diff --git a/pdf/pdf_interpret.c b/pdf/pdf_interpret.c
index b5938829..ea7f7692 100644
--- a/pdf/pdf_interpret.c
+++ b/pdf/pdf_interpret.c
@@ -942,6 +942,7 @@ copy_state(fz_context *ctx, pdf_gstate *gs, pdf_gstate *old)
 		pdf_keep_xobject(ctx, gs->softmask);
 }
 
+
 static pdf_csi *
 pdf_new_csi(pdf_document *xref, fz_device *dev, fz_matrix ctm, char *event, fz_cookie *cookie, pdf_gstate *gstate)
 {
@@ -2517,6 +2518,7 @@ pdf_run_keyword(pdf_csi *csi, fz_obj *rdb, fz_stream *file, char *buf)
 			fz_warn(ctx, "unknown keyword: '%s'", buf);
 		break;
 	}
+	fz_assert_lock_not_held(ctx, FZ_LOCK_FILE);
 }
 
 static void
diff --git a/pdf/pdf_repair.c b/pdf/pdf_repair.c
index 15886a8f..0dc0e132 100644
--- a/pdf/pdf_repair.c
+++ b/pdf/pdf_repair.c
@@ -67,7 +67,7 @@ pdf_repair_obj(fz_stream *file, char *buf, int cap, int *stmofsp, int *stmlenp,
 		}
 
 		obj = fz_dict_gets(dict, "Length");
-		if (fz_is_int(obj))
+		if (!fz_is_indirect(obj) && fz_is_int(obj))
 			stm_len = fz_to_int(obj);
 
 		fz_drop_obj(dict);
@@ -184,12 +184,14 @@ pdf_repair_obj_stm(pdf_document *xref, int num, int gen)
 				fz_throw(ctx, "corrupt object stream (%d %d R)", num, gen);
 		}
 	}
-	fz_catch(ctx)
+	fz_always(ctx)
 	{
 		fz_close(stm);
+	}
+	fz_catch(ctx)
+	{
 		fz_throw(ctx, "cannot load object stream object (%d %d R)", num, gen);
 	}
-	fz_close(stm);
 }
 
 void
@@ -386,7 +388,9 @@ pdf_repair_xref(pdf_document *xref, char *buf, int bufsize)
 			/* corrected stream length */
 			if (list[i].stm_len >= 0)
 			{
+				fz_unlock(ctx, FZ_LOCK_FILE);
 				dict = pdf_load_object(xref, list[i].num, list[i].gen);
+				fz_lock(ctx, FZ_LOCK_FILE);
 				/* RJW: "cannot load stream object (%d %d R)", list[i].num, list[i].gen */
 
 				length = fz_new_int(ctx, list[i].stm_len);
diff --git a/pdf/pdf_shade.c b/pdf/pdf_shade.c
index 15f5de6b..fb5dd72c 100644
--- a/pdf/pdf_shade.c
+++ b/pdf/pdf_shade.c
@@ -604,7 +604,7 @@ pdf_load_mesh_params(pdf_document *xref, fz_obj *dict, struct mesh_params *p)
 
 static void
 pdf_load_type4_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
-	int funcs, pdf_function **func, fz_stream *stream)
+	int funcs, pdf_function **func)
 {
 	fz_context *ctx = xref->ctx;
 	struct mesh_params p;
@@ -612,6 +612,7 @@ pdf_load_type4_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 	int ncomp;
 	int flag;
 	int i;
+	fz_stream *stream;
 
 	pdf_load_mesh_params(xref, dict, &p);
 
@@ -623,6 +624,8 @@ pdf_load_type4_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 	else
 		ncomp = shade->colorspace->n;
 
+	stream = pdf_open_stream(xref, fz_to_num(dict), fz_to_gen(dict));
+
 	while (!fz_is_eof_bits(stream))
 	{
 		flag = fz_read_bits(stream, p.bpflag);
@@ -665,11 +668,12 @@ pdf_load_type4_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 			break;
 		}
 	}
+	fz_close(stream);
 }
 
 static void
 pdf_load_type5_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
-	int funcs, pdf_function **func, fz_stream *stream)
+	int funcs, pdf_function **func)
 {
 	fz_context *ctx = xref->ctx;
 	struct mesh_params p;
@@ -677,6 +681,7 @@ pdf_load_type5_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 	int first;
 	int ncomp;
 	int i, k;
+	fz_stream *stream;
 
 	pdf_load_mesh_params(xref, dict, &p);
 
@@ -692,6 +697,8 @@ pdf_load_type5_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 	buf = fz_malloc_array(ctx, p.vprow, sizeof(struct vertex));
 	first = 1;
 
+	stream = pdf_open_stream(xref, fz_to_num(dict), fz_to_gen(dict));
+
 	while (!fz_is_eof_bits(stream))
 	{
 		for (i = 0; i < p.vprow; i++)
@@ -713,13 +720,14 @@ pdf_load_type5_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 
 	fz_free(ctx, ref);
 	fz_free(ctx, buf);
+	fz_close(stream);
 }
 
 /* Type 6 & 7 -- Patch mesh shadings */
 
 static void
 pdf_load_type6_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
-	int funcs, pdf_function **func, fz_stream *stream)
+	int funcs, pdf_function **func)
 {
 	fz_context *ctx = xref->ctx;
 	struct mesh_params p;
@@ -728,6 +736,7 @@ pdf_load_type6_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 	fz_point prevp[12];
 	int ncomp;
 	int i, k;
+	fz_stream *stream;
 
 	pdf_load_mesh_params(xref, dict, &p);
 
@@ -741,6 +750,8 @@ pdf_load_type6_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 
 	hasprevpatch = 0;
 
+	stream = pdf_open_stream(xref, fz_to_num(dict), fz_to_gen(dict));
+
 	while (!fz_is_eof_bits(stream))
 	{
 		float c[4][FZ_MAX_COLORS];
@@ -834,11 +845,12 @@ pdf_load_type6_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 			hasprevpatch = 1;
 		}
 	}
+	fz_close(stream);
 }
 
 static void
 pdf_load_type7_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
-	int funcs, pdf_function **func, fz_stream *stream)
+	int funcs, pdf_function **func)
 {
 	fz_context *ctx = xref->ctx;
 	struct mesh_params p;
@@ -847,6 +859,7 @@ pdf_load_type7_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 	fz_point prevp[16];
 	int ncomp;
 	int i, k;
+	fz_stream *stream;
 
 	pdf_load_mesh_params(xref, dict, &p);
 
@@ -860,6 +873,8 @@ pdf_load_type7_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 
 	hasprevpatch = 0;
 
+	stream = pdf_open_stream(xref, fz_to_num(dict), fz_to_gen(dict));
+
 	while (!fz_is_eof_bits(stream))
 	{
 		float c[4][FZ_MAX_COLORS];
@@ -953,6 +968,7 @@ pdf_load_type7_shade(fz_shade *shade, pdf_document *xref, fz_obj *dict,
 			hasprevpatch = 1;
 		}
 	}
+	fz_close(stream);
 }
 
 /* Load all of the shading dictionary parameters, then switch on the shading type. */
@@ -962,7 +978,6 @@ pdf_load_shading_dict(pdf_document *xref, fz_obj *dict, fz_matrix transform)
 {
 	fz_shade *shade = NULL;
 	pdf_function *func[FZ_MAX_COLORS] = { NULL };
-	fz_stream *stream = NULL;
 	fz_obj *obj;
 	int funcs = 0;
 	int type = 0;
@@ -972,7 +987,6 @@ pdf_load_shading_dict(pdf_document *xref, fz_obj *dict, fz_matrix transform)
 	fz_var(shade);
 	fz_var(func);
 	fz_var(funcs);
-	fz_var(stream);
 	fz_var(type);
 
 	fz_try(ctx)
@@ -1041,35 +1055,25 @@ pdf_load_shading_dict(pdf_document *xref, fz_obj *dict, fz_matrix transform)
 			}
 		}
 
-		if (type >= 4 && type <= 7)
-		{
-			stream = pdf_open_stream(xref, fz_to_num(dict), fz_to_gen(dict));
-			/* RJW: "cannot open shading stream (%d %d R)", fz_to_num(dict), fz_to_gen(dict) */
-		}
-
 		switch (type)
 		{
 		case 1: pdf_load_function_based_shading(shade, xref, dict, func[0]); break;
 		case 2: pdf_load_axial_shading(shade, xref, dict, funcs, func); break;
 		case 3: pdf_load_radial_shading(shade, xref, dict, funcs, func); break;
-		case 4: pdf_load_type4_shade(shade, xref, dict, funcs, func, stream); break;
-		case 5: pdf_load_type5_shade(shade, xref, dict, funcs, func, stream); break;
-		case 6: pdf_load_type6_shade(shade, xref, dict, funcs, func, stream); break;
-		case 7: pdf_load_type7_shade(shade, xref, dict, funcs, func, stream); break;
+		case 4: pdf_load_type4_shade(shade, xref, dict, funcs, func); break;
+		case 5: pdf_load_type5_shade(shade, xref, dict, funcs, func); break;
+		case 6: pdf_load_type6_shade(shade, xref, dict, funcs, func); break;
+		case 7: pdf_load_type7_shade(shade, xref, dict, funcs, func); break;
 		default:
 			fz_throw(ctx, "unknown shading type: %d", type);
 		}
 
-		if (stream)
-			fz_close(stream);
 		for (i = 0; i < funcs; i++)
 			if (func[i])
 				pdf_drop_function(ctx, func[i]);
 	}
 	fz_catch(ctx)
 	{
-		if (stream)
-			fz_close(stream);
 		for (i = 0; i < funcs; i++)
 			if (func[i])
 				pdf_drop_function(ctx, func[i]);
diff --git a/pdf/pdf_stream.c b/pdf/pdf_stream.c
index 3db8eb1c..a01bab55 100644
--- a/pdf/pdf_stream.c
+++ b/pdf/pdf_stream.c
@@ -225,6 +225,7 @@ pdf_open_filter(fz_stream *chain, pdf_document *xref, fz_obj *stmobj, int num, i
 	else if (fz_array_len(filters) > 0)
 		chain = build_filter_chain(chain, xref, filters, params, num, gen);
 
+	fz_lock_stream(chain);
 	return chain;
 }
 
@@ -276,6 +277,7 @@ pdf_open_raw_stream(pdf_document *xref, int num, int gen)
 		fz_throw(xref->ctx, "object is not a stream");
 
 	stm = pdf_open_raw_filter(xref->file, xref, x->obj, num, gen);
+	fz_lock_stream(stm);
 	fz_seek(xref->file, x->stm_ofs, 0);
 	return stm;
 }
@@ -316,6 +318,7 @@ pdf_open_stream_with_offset(pdf_document *xref, int num, int gen, fz_obj *dict,
 		fz_throw(xref->ctx, "object is not a stream");
 
 	stm = pdf_open_filter(xref->file, xref, dict, num, gen);
+	fz_lock_stream(stm);
 	fz_seek(xref->file, stm_ofs, 0);
 	return stm;
 }
@@ -378,9 +381,6 @@ pdf_load_stream(pdf_document *xref, int num, int gen)
 
 	fz_var(buf);
 
-	stm = pdf_open_stream(xref, num, gen);
-	/* RJW: "cannot open stream (%d %d R)", num, gen */
-
 	dict = pdf_load_object(xref, num, gen);
 	/* RJW: "cannot load stream dictionary (%d %d R)", num, gen */
 
@@ -393,16 +393,21 @@ pdf_load_stream(pdf_document *xref, int num, int gen)
 
 	fz_drop_obj(dict);
 
+	stm = pdf_open_stream(xref, num, gen);
+	/* RJW: "cannot open stream (%d %d R)", num, gen */
+
 	fz_try(ctx)
 	{
 		buf = fz_read_all(stm, len);
 	}
-	fz_catch(ctx)
+	fz_always(ctx)
 	{
 		fz_close(stm);
+	}
+	fz_catch(ctx)
+	{
 		fz_throw(ctx, "cannot read raw stream (%d %d R)", num, gen);
 	}
 
-	fz_close(stm);
 	return buf;
 }
diff --git a/pdf/pdf_xref.c b/pdf/pdf_xref.c
index 210c2f44..41a7ba3a 100644
--- a/pdf/pdf_xref.c
+++ b/pdf/pdf_xref.c
@@ -293,6 +293,8 @@ pdf_read_new_xref_section(pdf_document *xref, fz_stream *stm, int i0, int i1, in
 	}
 }
 
+/* Entered with file locked. Drops the lock in the middle, but then picks
+ * it up again before exiting. */
 static fz_obj *
 pdf_read_new_xref(pdf_document *xref, char *buf, int cap)
 {
@@ -319,6 +321,7 @@ pdf_read_new_xref(pdf_document *xref, char *buf, int cap)
 
 	fz_try(ctx)
 	{
+		fz_unlock(ctx, FZ_LOCK_FILE);
 		obj = fz_dict_gets(trailer, "Size");
 		if (!obj)
 			fz_throw(ctx, "xref stream missing Size entry (%d %d R)", num, gen);
@@ -368,10 +371,12 @@ pdf_read_new_xref(pdf_document *xref, char *buf, int cap)
 		fz_drop_obj(index);
 		fz_rethrow(ctx);
 	}
+	fz_lock(ctx, FZ_LOCK_FILE);
 
 	return trailer;
 }
 
+/* File is locked on entry, and exit (but may be dropped in the middle) */
 static fz_obj *
 pdf_read_xref(pdf_document *xref, int ofs, char *buf, int cap)
 {
@@ -649,10 +654,12 @@ pdf_open_document_with_stream(fz_stream *file)
 	fz_obj *obj;
 	fz_obj *nobj = NULL;
 	int i, repaired = 0;
+	int locked;
 	fz_context *ctx = file->ctx;
 
 	fz_var(dict);
 	fz_var(nobj);
+	fz_var(locked);
 
 	/* install pdf specific callback */
 	fz_resolve_indirect = pdf_resolve_indirect;
@@ -663,6 +670,9 @@ pdf_open_document_with_stream(fz_stream *file)
 	xref->file = fz_keep_stream(file);
 	xref->ctx = ctx;
 
+	fz_lock(ctx, FZ_LOCK_FILE);
+	locked = 1;
+
 	fz_try(ctx)
 	{
 		pdf_load_xref(xref, xref->scratch, sizeof xref->scratch);
@@ -691,6 +701,9 @@ pdf_open_document_with_stream(fz_stream *file)
 		if (repaired)
 			pdf_repair_xref(xref, xref->scratch, sizeof xref->scratch);
 
+		fz_unlock(ctx, FZ_LOCK_FILE);
+		locked = 0;
+
 		encrypt = fz_dict_gets(xref->trailer, "Encrypt");
 		id = fz_dict_gets(xref->trailer, "ID");
 		if (fz_is_dict(encrypt))
@@ -749,6 +762,11 @@ pdf_open_document_with_stream(fz_stream *file)
 			}
 		}
 	}
+	fz_always(ctx)
+	{
+		if (locked)
+			fz_unlock(ctx, FZ_LOCK_FILE);
+	}
 	fz_catch(ctx)
 	{
 		fz_drop_obj(dict);
@@ -910,18 +928,17 @@ pdf_load_obj_stm(pdf_document *xref, int num, int gen, char *buf, int cap)
 			}
 		}
 	}
-	fz_catch(ctx)
+	fz_always(ctx)
 	{
 		fz_close(stm);
 		fz_free(xref->ctx, ofsbuf);
 		fz_free(xref->ctx, numbuf);
 		fz_drop_obj(objstm);
+	}
+	fz_catch(ctx)
+	{
 		fz_throw(ctx, "cannot open object stream (%d %d R)", num, gen);
 	}
-	fz_close(stm);
-	fz_free(xref->ctx, ofsbuf);
-	fz_free(xref->ctx, numbuf);
-	fz_drop_obj(objstm);
 }
 
 /*
@@ -950,6 +967,7 @@ pdf_cache_object(pdf_document *xref, int num, int gen)
 	}
 	else if (x->type == 'n')
 	{
+		fz_lock(ctx, FZ_LOCK_FILE);
 		fz_seek(xref->file, x->ofs, 0);
 
 		fz_try(ctx)
@@ -959,6 +977,7 @@ pdf_cache_object(pdf_document *xref, int num, int gen)
 		}
 		fz_catch(ctx)
 		{
+			fz_unlock(ctx, FZ_LOCK_FILE);
 			fz_throw(ctx, "cannot parse object (%d %d R)", num, gen);
 		}
 
@@ -966,11 +985,13 @@ pdf_cache_object(pdf_document *xref, int num, int gen)
 		{
 			fz_drop_obj(x->obj);
 			x->obj = NULL;
+			fz_unlock(ctx, FZ_LOCK_FILE);
 			fz_throw(ctx, "found object (%d %d R) instead of (%d %d R)", rnum, rgen, num, gen);
 		}
 
 		if (xref->crypt)
 			pdf_crypt_obj(ctx, xref->crypt, x->obj, num, gen);
+		fz_unlock(ctx, FZ_LOCK_FILE);
 	}
 	else if (x->type == 'o')
 	{
diff --git a/scripts/cmapdump.c b/scripts/cmapdump.c
index 137e9abb..2f3c16df 100644
--- a/scripts/cmapdump.c
+++ b/scripts/cmapdump.c
@@ -49,7 +49,7 @@ main(int argc, char **argv)
 		return 1;
 	}
 
-	ctx = fz_new_context(NULL, FZ_STORE_UNLIMITED);
+	ctx = fz_new_context(NULL, NULL, FZ_STORE_UNLIMITED);
 	if (!ctx)
 	{
 		fprintf(stderr, "cannot initialise context\n");
@@ -203,6 +203,15 @@ int fz_store_scavenge(fz_context *ctx, unsigned int size, int *phase)
 	return 0;
 }
 
+void fz_new_glyph_cache_context(fz_context *ctx)
+{
+}
+
 void fz_free_glyph_cache_context(fz_context *ctx)
 {
 }
+
+fz_glyph_cache *fz_glyph_cache_keep(fz_context *ctx)
+{
+	return NULL;
+}
diff --git a/xps/xps_zip.c b/xps/xps_zip.c
index b3772138..08d9832a 100644
--- a/xps/xps_zip.c
+++ b/xps/xps_zip.c
@@ -91,12 +91,17 @@ xps_read_zip_entry(xps_document *doc, xps_entry *ent, unsigned char *outbuf)
 	int version, general, method;
 	int namelength, extralength;
 	int code;
+	fz_context *ctx = doc->ctx;
 
+	fz_lock(ctx, FZ_LOCK_FILE);
 	fz_seek(doc->file, ent->offset, 0);
 
 	sig = getlong(doc->file);
 	if (sig != ZIP_LOCAL_FILE_SIG)
+	{
+		fz_unlock(ctx, FZ_LOCK_FILE);
 		fz_throw(doc->ctx, "wrong zip local file signature (0x%x)", sig);
+	}
 
 	version = getshort(doc->file);
 	general = getshort(doc->file);
@@ -132,23 +137,32 @@ xps_read_zip_entry(xps_document *doc, xps_entry *ent, unsigned char *outbuf)
 
 		code = inflateInit2(&stream, -15);
 		if (code != Z_OK)
+		{
+			fz_unlock(ctx, FZ_LOCK_FILE);
 			fz_throw(doc->ctx, "zlib inflateInit2 error: %s", stream.msg);
+		}
 		code = inflate(&stream, Z_FINISH);
 		if (code != Z_STREAM_END)
 		{
 			inflateEnd(&stream);
+			fz_unlock(ctx, FZ_LOCK_FILE);
 			fz_throw(doc->ctx, "zlib inflate error: %s", stream.msg);
 		}
 		code = inflateEnd(&stream);
 		if (code != Z_OK)
+		{
+			fz_unlock(ctx, FZ_LOCK_FILE);
 			fz_throw(doc->ctx, "zlib inflateEnd error: %s", stream.msg);
+		}
 
 		fz_free(doc->ctx, inbuf);
 	}
 	else
 	{
+		fz_unlock(ctx, FZ_LOCK_FILE);
 		fz_throw(doc->ctx, "unknown compression method (%d)", method);
 	}
+	fz_unlock(ctx, FZ_LOCK_FILE);
 }
 
 /*
@@ -221,7 +235,9 @@ xps_find_and_read_zip_dir(xps_document *doc)
 	unsigned char buf[512];
 	int file_size, back, maxback;
 	int i, n;
+	fz_context *ctx = doc->ctx;
 
+	fz_lock(ctx, FZ_LOCK_FILE);
 	fz_seek(doc->file, 0, SEEK_END);
 	file_size = fz_tell(doc->file);
 
@@ -237,6 +253,7 @@ xps_find_and_read_zip_dir(xps_document *doc)
 			if (!memcmp(buf + i, "PK\5\6", 4))
 			{
 				xps_read_zip_dir(doc, file_size - back + i);
+				fz_unlock(ctx, FZ_LOCK_FILE);
 				return;
 			}
 		}
@@ -244,6 +261,7 @@ xps_find_and_read_zip_dir(xps_document *doc)
 		back += sizeof buf - 4;
 	}
 
+	fz_unlock(ctx, FZ_LOCK_FILE);
 	fz_throw(doc->ctx, "cannot find end of central directory");
 }