mupdf - MuPDF PDF reader and library

Age	Commit message (Collapse)	Author
2018-01-04	Add portable pseudo-random number generator based on the lrand48 family.	Tor Andersson

2017-11-01	Use int64_t for public file API offsets.	Tor Andersson
	Don't mess with conditional compilation with LARGEFILE -- always expose 64-bit file offsets in our public API.
2017-07-19	Add spots to fz_pixmaps.	Robin Watts
	Update separations interface further to cope with whether spots should be rendered separately, or as composite colors.
2017-06-22	Add Color Management interface to MuPDF, with LCMS implementation.	Michael Vrhel
	This commit adds an interface for a color management engine to MuPDF, together with the internal changes required to use it. We also add an implementation of this interface using the LCMS2 library. This commit is the "lcms" development branch squashed to a single commit. The individual commits that make it up are as follows: ------ Add LCMS2 submodule. Add required source files to MSVC libthirdparty project. Plus changes to the Makefile. ------ Change name of libfont to libresources The library can hold things other than fonts including ICC profiles and eventually halftone screens etc. ------ Generate and include icc profiles on windows solution Makefile for linux needs to be updated ------ Initial cut at interface to little cms Methods for getting profile handles, link handles and transform buffers as well as individual colors. ------ Install ICC profiles from source When the source color space is an ICC profile install the ICC space. Use alternative color space if ICC is invalid. ------ Rename ICC resources The way that fontdump generates the names was causing some redundant prefix/suffix combinations. ------ Make resource ICC profiles usable This brings in the resource profiles for use with the target device. When mudraw is invoked with the icc color type, it will set up the target pixmap to have an icc based color space. ------ Make Default ICC color spaces not storable The ICC color spaces in the color context should not be put into the store. ICC color spaces that are contained in the source document are however storable. ------ Add in support for single color conversion This adds in the selection of the icc single color converter. Note that we may want to look at using the float conversion in lcms here since we are going from and to float. The down side is that the creation of such converters may be expensive. ------ Pixmap ICC conversion Also clean up and further simplify some of the code. Use common dst, src in function parameters instead of src, dst. ------ Add md5 calculation for profiles Compare md5 of source and destination. If they are the same set link as identity and do not bother creating link from cmm. ------ Initial attempt at adding links to store Next need to make sure I have all the dropping set up correctly and that the links are getting freed when we are all done with them. ------ Add drop for link Make sure that we drop the links when we are done with them. ------ Fix icc link store ref. counting + rendering intent The key allocations, rc and removal was wrong for the icc links. Also added in support on the graphic state for rendering intent. ------ Add ICC profiles to Makefile. ------ Move ICC profile loading to colorspace.c ------ Fix build on linux. ------ Use hexdump in creating icc profiles ------ First cut at CalRGB CalGray handling These color spaces are converted to equivalent ICC profiles when the drawing operation occurs. ToDo still is avoid creating brand new CalGray, CalRGB or ICC color spaces when we encounter the same object in the source file. Instead we should make use of the store. ------ Adding fz_color_params into device API Stroke and fill of paths and text which will have color rendering related settings including overprint, overprint mode, rendering intent and black point settings. Images have a fz_color_params member added to them. ------ Rendering intent support for graphic and text Graphic fills and strokes as well as text fill and strokes handle the rendering intent settings. This works through the display list. The parameters related to color rendering, ri, bp, op, opm are packed in the flags bits through the display list. To do: Add support for images and shadings. ------ Add support for rendering intent with images Required change to fill_image api, hence the large number of files touched. ------ Add support for shadings rendering intent and DeviceN This adds support for the rendering intent in shading fills. Also, adds in support for color management of DeviceN and Separation color spaces where the base space is defined to be ICC. ------ Add clamping proc to colorspace-imp.h In the head of mupdf, the index color values were not being clamped properly. I moved the clamping operation to be performed by a procedure in of the color space. The a, b values in LAB will need to include a range value for proper clamping as well as the ICC color space at some point. ------ Fix assert test with index images ------ Support for DeviceN images color managed The base space for DeviceN and separation images is now color managed when the ICC work flow is used. ------ Add DefaultGray etc to names.txt The DefaultGray, DefaultRGB, and DefaultCMYK settings in the pdf page resource dictionary need to be parsed and handled if present. ------ Remove methods for setting color space in context The methods that existed for setting the color spaces in the color space context were broken and we will not be changing them from their initial setting at startup. ------ Add front end support for default color spaces PDF can specify in the resource dictionary color spaces that should be used for DeviceGray, DeviceRGB and DeviceCMYK. This commit handles the extraction of these color spaces and passes them through the display list and gets them into a structure on a draw device (if one is used). Next step is to have the draw device make use of them. ------ Backend of default color spaces This handles the use of page level definitions for DefaultGray, DefaultRGB, DefaultCMYK in the draw device. The interface for the pixmap color conversion had to be expanded a bit to pass the default_cs object. This was needed due to the fact that the image could be DeviceN or Separation and have a base space that needs to be replaced by a page level Default color space. Tested this with a file that had two pages each with an common reference image object but different DefaultCMYK definitions in the page resource dictionary. Proper rendering was obtained. ------ Add icc support to band-writer and png output For image viewers that support icc profiles, it is important that we include the icc profile into those formats that support. Currently the only output format we have that supports it is PNG. But if we decide to do tiff, jpeg or psd those also support the embedding of icc profiles. ------ Work toward multithreading Each cloned context creates a new cmm context to reduce contention. This may not be optimal though as it may create conflicts in any shared store. ------ Add missing files. Failed to check in icc-imp.h and icc34.h ------ Fix default device color calls copy and paste errors. ------ Fixes ------ Fix clamping of lab colors The lab colors when used with the lab resource profile need to be properly clamped and scaled. ------ Turn off ICC create debug define ------ Fix memory leak issues ------ Set clamp proc in color context color spaces When NO_ICC is enabled there was an issue as the default spaces did not have the clamp proc (which was added in the lcms branch). ------ Fix several issues after rebase During rebase of the branch there were several issues that came up. In addition there was a fix related to the use of the lcms2 context branch that did not make it in for some reason. ------ Fix for fz_store_type structure changes ------ Fixes for multithreaded This fixes a few issues. One issue is that we can not be changing the input and output formatter for the links if they are being shared among threads. This adds the format as part of the link definition. Next step is to create a link clone operation in lcms so that we can readily create a different one with everything the same except the formatter. Also, this shares the profiles that exist in the color context among the threads. When a context is cloned, it will use the profiles in the current context but it will create a new cmm context.c ------ Fix NO_ICC issues There were a few issues that occurred when we compiled and ran with the NO_ICC setting. ------ Change CMM to use lower resolution tables in links The table resolution greatly affects the performance for some files. I don't see any significant color rendering issues going with a lower resolution. Selecting cmsFLAGS_LOWRESPRECALC uses a MLUT resolution of 17. This is going to be sufficient color wise and gives a large improvement performance wise especially for files where the color link creation is significant compared to the rendering. ------ Fix link key creation to populate alpha and bit depth. The store key for icc links needs to include the bit depth and if the transform has to handle an alpha value. This is needed due to the fact that we can't change the formatter when we are sharing links among different threads. ------ Pull in some MSVC 2005 fixes for lcms2 ------ Fix non-prototype prototype. ------ Miscellaneous typos, whitespace fixes. ------ Tidy colorspace creation API. Rather than pass a magic implementation reference count value (-1 for static, 1 for normal), pass a boolean "is_static" flag. This gives a nicer API (IMAO etc). ------ Make all colorspaces use an fz_buffer. ------ Fix internal naming of MSVC project file. ------ Fix some error handling. ------ Add some consts. ------ Tweak fz_color_params and fz_store_hash. Use uint8_t for fz_color_param entries - no need to use larger. Same change in fz_store_hash. This limits the size required for the hash table too. ------ Throw on errors, rather than returning NULL. fz_cmm_transform_pixmap should throw if it's fed pixmaps incompatible with its transforms. fz_cmm_new_ctx should throw if the cmm fails to initialise. ------ Ensure LCMS2_OBJs are built/linked using Makefile. Also ensure that ICC_OBJs are included. ------ Fix some unused variable warnings. ------ cs_params fixes. Use the ones that are passed. ------ fz_color_params tweaks. Make them const in many places. Cope with cs_params == NULL meaning default (i.e. fz_cs_param(ctx)). Minimise the places we call fz_cs_params(ctx). Consistently use cs_params rather than a mix of cs_param and cs_params. ------ Improve PDF color params handling. PDF allows different OP settings for strokes and fills. This means that either we need to keep separate entries in each fz_color_param structure for the stroke one or the fill one, OR we need to duplicate the fz_color_param structure. It seems neatest to do the latter (not least because this means we don't pass more information to each device function than it actually needs). Accordingly, we put an fz_color_param in each pdf_material. We update the code that reads ExtGStates to set the values appropriately. We take the opportunity to add support for the PDF 2.0 UseBlackPtComp option too. ------ Fix colorspace ref count problems. 1) Don't drop colorspace until we've finished using it. 2) Don't drop it twice. ------ Convert NULL deref into a thrown error. Seen with: mutool draw -D -o out.png ../tests_private/comparefiles/Bug689760.pdf We seem to have a pdf-cal space with no profile. Talk to Michael about this. ------ Avoid using colorspace names to distinguish colorspaces. Using strcmp is slow. ------ Cope with failure to parse default colorspaces during PDF page load. ------ Avoid SEGV due to to pdf_cal space with no profile. As seen in: tests_private/pdf/PDFIA1.7_SUBSET/CATX4879.pdf ------ Handle cases where base space of sep is pdf-cal We handled this for images, but not for solid fills. ------ Accelerate ICC color conversion. ------ Cope with indexed images in the color management. Some images (JPX images) come to the cm code still in the indexed color space. Their base space could be DevN so we need to cope with multiple base decodes. This continues the decode until we get to a space for which we can create an ICC link. ------ Ignore alpha presence in component count check ------ Eliminate recursion in fz_source_colorspace_cm. ------ Cope with bare ICCBased colorspace defs. Bug692137.pdf has ICCBased colorspaced definitions given directly as a stream, rather than as [ /ICCBased <stream reference> ] Acrobat copes with these, as does gs. We therefore update our code to cope too. Also, the PDF spec says that any problems found when reading the Default spaces should be ignored (or at least not to abort rendering). Update our code to do that too. ------ Tweak color converter logic for speed. When we are using an icc profile based conversion, avoid ever having to lookup the color converter for each conversion call; do it at the lookup stage. ------ Harden us against failures during ICC link creation. Seen with corrupted profiles. ------ Fix for handling alpha with lcms. Note that this currently maintains an alpha when it is present. We may need to do some work for the gray scale conversion to alpha mask. ------ Delay pdf-cal profile creation Put the creation of the pdf cal profile into the link creation function (rather than have it scattered). Also be robust in the condition of failure to create the profile. ------ Proper clamping with embedded CIELAB ICC profile If the ICC profiles alternate color space is LAB then use the LAB clamping proc. ------ Use the color space clamp when converting to base color spaces In many of the Ghent test files, we have a DeviceN image whose alternate color space is CIELAB. We need to make sure to use the CIELAB clamping operation in this case. ------ Make ICC runtime configurable. Add fz_icc_flowflow, fz_set_icc_workflow functions to read ICC workflow state, and set the ICC workflow state. The latter will throw an error if trying to enable ICC workflow in a NO_ICC build. Add -N flag to mutool to disable ICC. Incorporates build fixes from Michael. ------ Ensure fz_draw_fill_image uses the pixmap colorspace It appears that the image colorspace historically has been able to differ from the pixmap colorspace. While we've fixed the cases that we know about for this (see the previous commits), tweak fz_draw_fill_image to work the way it always has in the past. ------ Add support for output intent PDF documents can have an ICC profile defined in their Catalog which defines the output color space and the color space to use for one of the Device color spaces (e.g. DeviceGray, DeviceRGB or DeviceCMYK). ------ Catch errors in default color spaces Before setting the default color space contained in the file, make sure it is the correct type. Bug692213.pdf ------ Clamp to base space during sep color conversion This was the source of a problem when the base space was CIELAB. ------ Rename some functions to be more MuPDFy drop rather than free etc. ------ Fix "casting away const" warning. srcv is a const *. Do the clamping operation on src_map (the same value) before it is assigned into the const variable. ------ Rejig top level color management interface slightly. Same code, just change the encapsulation. ------ Remove pre-multiplied alpha prior to color management The pixmaps in mupdf use a premultiplied alpha format. Prior to doing any color management we need to undo the alpha and then reapply after color management. ------ Remove global output intent as unused. ------ Move fz_color_params to be the final arg in dev calls. Frequently this will be NULL, and it doesn't form part of the colorspace/color/alpha triple. ------ Rename fz_default_xxx static variables. Remove fz prefix, to prepare for a later renaming that would conflict. ------ Rename cs_params to color_params. ------ Rename lcms branch identifiers. ------ Return device colorspaces if the default colorspace is NULL. ------ Clean up device call function for set_default_colorspaces. ------ Add missing rethrow. ------ Load page default colorspaces lazily. ------ murun: Add color params device argument. Stubbed to always be NULL at the moment. ------ Rename extgstate processor ops. ------ Fix a few minor issues from Tor Removes icc-imp.h Rename color-icccreate.c Add context to some methods ------ Fix javalib with recent lcms dev changes. ------ Update lcms2 with sub project fixes. ------ Fix build failure. ------ Add icc profile into other PNG output methods ------ Fix some ints that should be size_t's. ------ Tweak fz_new_icc_data_from_icc_colorspace. Ensure that the colorspace is const, and that we set size to 0 if we don't find any data. ------ Combine band writer methods for header and ICC writing For many formats (like PSD), we need to delay writing some of the header until we know whether we are getting an ICC profile to write or not. Makes more sense to just write them at the same time. ------ Miscellaneous tweaks in colorspace.c Mainly to avoid pointer aliasing, "nicer" whitespace, and a leak on error. ------ Avoid rightward drift in get_base_icc. ------ Ensure that pdf_load_output_intent copes with exceptions. If load_icc_based throws and exception, warn and continue. Don't not render a file just because the inbuilt default profiles are broken. ------ Revert change in pdf-page.c The page resources load had been moved so we could get the default colorspaces, but this has been moved into run_contents, so putting stuff back as it was before. ------ Revert changes in mudraw. We had added some colorspages in mudraw, and then later removed them. Revert the changes to accomodate this to make the overall branch diffs smaller. ------ Use fz_buffer in color-icc-create.c ------ Force mapping through proof icc profile always. ------ Fix behaviour on fz_cmm_new_profile failure. If the profile fails to be made, return an error code, and have the callers take sane steps. ------ Tweak load_icc_based Cope better with errors in reading the ICC space not stopping us loading the alternate. ------ Remove unused variable. ------ Get page resource for DefaultCS look-up res = pdf_dict_get(ctx, PDF_NAME_Resources, page->obj); was not returning resource. Replaced with res = pdf_page_resources(ctx, page); ------ Move default color space set up to pdf_run_page_contents_with_usage ------ Review fixes for lcms branch. ------ Fixes for calibrated colorspace loading. ------ Add fz_document_output_intent wrapper. Lazy load intent for PDF. ------ Copy DefaultCS logic into pdf_run_annot_with_usage. Same code as from pdf_run_page_contents_with_usage. ------ More review fixes. ------ Avoid rightward drift in pdf_load_cal_common ------ Rename color_converter functions to be find/drop. Better than than lookup/discard. lookup suggests something that doesn't need dropping, and we use drop rather than discard by convention. ------ Move cmm from context into colorspace context. ------ Review fixes: Remove recursion and rename functions. ------ Don't access doc->oi directly in pdf_load_default_colorspaces. ------ Rename fz_colorspace_is_pdf_cal to fz_colorspace_is_cal and make it public. ------ Tweak function naming to be more consistent. ------ fz_md5_icc can be implemented using fz_md5_buffer. ------ Print full md5 checksums in link key debug prints. ------ Make fz_md5_buffer NULL safe. ------ Simplify debug saving of ICC profiles. ------ Rename fz_cmm_new/drop_profile to init/fin. ------ Indentation cleanups. ------ Move CMM static inline functions from private header to C file. ------ Tweak fz_icc_data_from_icc_colorspace to return a buffer. Also, remove the _new_ from the name to reflect the fact that we are passed a borrowed handle, not given a new reference. ------ java: Add ColorParams.pack() function. ------ Generate one C file for the embedded ICC profiles. ------ Return const pointer from fz_default_color_params. ------ Change misleading argument names to fz_new_colorspace. ------ Rename fz_cmm_new/drop_link to fz_cmm_init/fin_link. ------ Change definition of fz_cmm_instance. Rather than void, use an undefined struct in keeping with the rest of the code. ------ Add support for color managed bgr color space ------ Return unsigned char array from fz_lookup_icc. ------ Make default_color_params immutable. Changing the defaults used by the draw device should happen via a device call, should we need the functionality in the future. ------ Clean up error handling in color-lcms.c ------ Fix signed/unsigned warning.
2017-05-23	Remove unused strtod function.	Tor Andersson

2017-04-27	Include required system headers.	Tor Andersson

2017-04-18	Use splay trees for loading/merging cmaps.	Robin Watts
	This allows for overlaps, merges adjacent (mergeable) ranges and gets us properly searchable results. This causes 1 diff in the test suites (Bug694353.pdf), which is due to the fallback font not having a hypen present at UCS 0x2010.
2017-04-13	Change generated files to be C instead of headers.	Tor Andersson
	Remove AdobeCA.p7c from autogenerated files. Just include the array in the source. Simplifies makefile dependencies and makes the sizes of each bit of data easier to look at. It also paves the way for eventually using objcopy to create binary objects for the fonts instead of needing to use hexdump.
2016-09-01	Simplify PDF resource caching table handling.	Tor Andersson

2016-06-08	Move to using size_t for all mallocs.	Robin Watts
	This has knock on effects in the store. fix
2016-05-13	Introduce a general output context.	Sebastian Rasmussen
	This makes it possible to redirect standard out and standard error output streams to output streams of your liking. This means that now you can, in gdb, type: (gdb) call pdf_print_obj(ctx, fz_stdout(ctx), obj, 0) (gdb) call fflush(0) or when dealing with an unresolved indirect reference: (gdb) call pdf_print_obj(ctx, fz_stdout(ctx), pdf_resolve_indirect(ctx, ref), 0) (gdb) call fflush(0)
2016-04-28	Introduce tuning context.	Robin Watts
	For now, just use it for controlling image decoding and image scaling.
2016-02-10	Add build=sanitize option to makefile.	Tor Andersson

2016-02-03	Bug 696546: Add fast strtof	Robin Watts
	Take on a (slightly tweaked) version of Simon Reinhardt's patch. The actual logic is left entirely unchanged; minor changes have been made to the names of functions/types to avoid clashing in the cmapdump.c repeated inclusion. Currently this should really only affect xps files, as strtof is only used as fz_atof, and that's (effectively) all xps for now. I will look at updating lex_number to call this in future.
2016-01-13	Add lots of consts.	Robin Watts
	In general, we should use 'const fz_blah' in device calls whenever the callee should not alter the fz_blah. Push this through. This shows up various places where we fz_keep and fz_drop these const things. I've updated the fz_keep and fz_drops with appropriate casts to remove the consts. We may need to do the union dance to avoid the consts for some compilers, but will only do that if required. I think this is nicer overall, even allowing for the const<->no const problems.
2015-12-14	Fix Windows build; cmapdump.c requires fz_fopen_utf8	Robin Watts

2015-05-15	Support pdf files larger than 2Gig.	Robin Watts
	If FZ_LARGEFILE is defined when building, MuPDF uses 64bit offsets for files; this allows us to open streams larger than 2Gig. The downsides to this are that: * The xref entries are larger. * All PDF ints are held as 64bit things rather than 32bit things (to cope with /Prev entries, hint stream offsets etc). * All file positions are stored as 64bits rather than 32. The implementation works by detecting FZ_LARGEFILE. Some #ifdeffery in fitz/system.h sets fz_off_t to either int or int64_t as appropriate, and sets defines for fz_fopen, fz_fseek, fz_ftell etc as required. These call the fseeko64 etc functions on linux (and so define _LARGEFILE64_SOURCE) and the explicit 64bit functions on windows.
2015-02-17	Add ctx parameter and remove embedded contexts for API regularity.	Tor Andersson
	Purge several embedded contexts: Remove embedded context in fz_output. Remove embedded context in fz_stream. Remove embedded context in fz_device. Remove fz_rebind_stream (since it is no longer necessary). Remove embedded context in svg_device. Remove embedded context in XML parser. Add ctx argument to fz_document functions. Remove embedded context in fz_document. Remove embedded context in pdf_document. Remove embedded context in pdf_obj. Make fz_page independent of fz_document in the interface. We shouldn't need to pass the document to all functions handling a page. If a page is tied to the source document, it's redundant; otherwise it's just pointless. Fix reference counting oddity in fz_new_image_from_pixmap.
2015-02-17	Rename fz_close_* and fz_free_* to fz_drop_*.	Tor Andersson
	Rename fz_close to fz_drop_stream. Rename fz_close_archive to fz_drop_archive. Rename fz_close_output to fz_drop_output. Rename fz_free_* to fz_drop_. Rename pdf_free_ to pdf_drop_. Rename xps_free_ to xps_drop_*.
2014-09-22	Fix 695467: Add and use fz_ftoa function (like dtoa but with floats).	Tor Andersson
	The dtoa function is for doubles (which is what MuJS uses) but for MuPDF we only need and want float precision in our output formatting.
2014-09-02	Add locale-independent number formatting and parsing functions.	Tor Andersson

2014-06-26	Fix some compiler warnings on Android.	Matt Holgate
	Use intptr_t when casting between a jlong and a pointer to suppress errors about different size words. Add a 'u' suffix to unsigned values output by the cmap dump utility.
2014-05-10	Fix 694698: Support 32-bit values in CMaps.	Tor Andersson
	Increasing the existing data structure to 32-bit values would bloat the data tables too much. Simplify the data structure and use three separate range tables for lookups -- one with small 16-bit to 16-bit range lookups, one with 32-bit range lookups, and a final one for one-to-many lookups. This loses the range-to-table optimization we had before, but even with the extra ranges this necessitates, the total size of the compiled binary CMap data is smaller than if we were to extend the previous scheme to 32 bits.
2014-03-19	Implement our own vsnprintf variant.	Tor Andersson
	The primary motivator for this is so that we can print floating point values and get the full accuracy out, without having to print 1.5 as 1.5000000, and without getting 23e24 etc. We only support %c, %f, %d, %o, %x and %s currently. We only support the zero padding qualifier, for integers. We do support some extensions: %C turns values >=128 into UTF-8. %M prints a fz_matrix. %R prints a fz_rect. %P prints a fz_point. We also implement a fprintf variant on top of this to allow for consistent results when using fz_output. a
2014-01-07	Introduce 'document handlers'.	Robin Watts
	We define a document handler for each file type (2 in the case of PDF, one to handle files with the ability to 'run' them, and one without). We then register these handlers with the context at startup, and then call fz_open_document... as usual. This enables people to select the document types they want at will (and even to extend the library with more document types should they wish).
2013-08-28	make cmapdump ignore VCS folders	Simon Bünzli
	If MuPDF is used in a project using Subversion or another VCS adding hidden subfolders to each folder, cmapdump breaks when trying to load the subfolder as cmap file. This fix is required starting with 643370f04348569b5e5e577660031d638537671c
2013-06-20	Update source, makefiles and win32 projects.	Tor Andersson

2013-06-18	Merge common and internal headers into one.	Tor Andersson

2013-06-18	Move header files into separate include directory.	Tor Andersson

2013-05-27	Strip trailing whitespace.	Tor Andersson

2013-05-17	Add colorspace context dummy functions to cmapdump.c	Tor Andersson

2013-03-01	Bug 693624: Ensure that windows copes with utf8 filenames	Robin Watts
	When running under Windows, replace fopen with our own fopen_utf8 that converts from utf8 to unicode before calling the unicode version of fopen.
2012-06-25	Warning fixes and various clean ups:	Sebastian Rasmussen
	Remove unused variable, silencing compiler warning. No need to initialize variables twice. Remove initialization of unread variable. Remove unnecessary check for NULL. Close output file upon error in cmapdump.
2012-05-08	Switch to reading content streams on the fly during interpretation.	Robin Watts
	Previously, before interpreting a pages content stream we would load it entirely into a buffer. Then we would interpret that buffer. This has a cost in memory use. Here, we update the code to read from a stream on the fly. This has required changes in various different parts of the code. Firstly, we have removed all use of the FILE lock - as stream reads can now safely be interrupted by resource (or object) reads from elsewhere in the file, the file lock becomes a very hard thing to maintain, and doesn't actually benefit us at all. The choices were to either use a recursive lock, or to remove it entirely; I opted for the latter. The file lock enum value remains as a placeholder for future use in extendable data streams. Secondly, we add a new 'concat' filter that concatenates a series of streams together into one, optionally putting whitespace between each stream (as the pdf parser requires this). Finally, we change page/xobject/pattern content streams to work on the fly, but we leave type3 glyphs using buffers (as presumably these will be run repeatedly).
2012-04-09	Bug 692977: Stop harmless thread debugging messages during cmapdump	Robin Watts
	If compiled with -DDEBUG, cmapdump throws a large number of warnings regarding thread locking. These are harmless and can be ignored, but are, nonetheless, not pretty. Fixed here. Thanks to Bas Weelinck for the report.
2012-03-12	More API tidying.	Robin Watts
	Make fz_clone_context copy existing AA settings. Add accessor function for fz_bitmap. Add more documentation for various functions/types.
2012-03-06	Split fitz.h/mupdf.h into internal/external headers.	Robin Watts
	Attempt to separate public API from internal functions.
2012-03-01	Add some more docs to fitz.h	Robin Watts
	Add docs for fz_store, fz_image, fz_halftones. Move fz_item definition into res_store.c as it does not need to be external. Rename fz_store_context to fz_keep_store_context to be consistent.
2012-02-13	Add locking around freetype calls.	Robin Watts
	We only open one instance of freetype per document. We therefore have to ensure that only 1 call to it takes place at a time. We introduce a lock for this purpose (FZ_LOCK_FREETYPE), and arrange to take/release it as required. We also update the font context so it is properly shared.
2012-02-08	Lock reworking.	Robin Watts
	This is a significant change to the use of locks in MuPDF. Previously, the user had the option of passing us lock/unlock functions for a single mutex as part of the allocation struct. Now we remove these entries from the allocation struct, and make a separate 'locks' struct. This enables people to use fz_alloc_default with locking. If multithreaded operation is required, then the user is required to create FZ_LOCK_MAX mutexes, which will be locked or unlocked by MuPDF calling the lock/unlock functions within the new fz_locks_context structure passed in at context creation. These mutexes are not required to be recursive (they may be, but MuPDF should never call them in this way). MuPDF avoids deadlocks by imposing a locking ordering on itself; a thread will never take lock n, if it already holds any lock i for which 0 <= i <= n. Currently, there are 4 locks used within MuPDF. Lock 0: The alloc lock; taken around all calls to user supplied (or default) allocation functions. Also taken around all accesses to the refs field of storable items. Lock 1: The store lock; taken whenever the store data structures (specifically the linked list pointers) are accessed. Lock 2: The file lock; taken whenever a thread is accessing the raw file. We use the debugging macros to insist that this is held whenever we do a file based seek or read. We also insist that this is never held when we resolve an indirect reference, as this can have the effect of moving the file pointer. Lock 3: The glyphcache lock; taken whenever a thread calls freetype, or accesses the glyphcache data structures. This introduces some complexities w.r.t type3 fonts. Locking can be hugely problematic, so to ease our minds as to the correctness of this code, we introduce some debugging macros. These compile away to nothing unless FITZ_DEBUG_LOCKING is defined. fz_assert_lock_held(ctx, lock) checks that we hold lock. fz_assert_lock_not_held(ctx, lock) checks that we do not hold lock. In addition fz_lock_debug_lock and fz_lock_debug_unlock are used on every fz_lock/fz_unlock to check the validity of the operation we are performing - in particular it checks that we do/do not already hold the lock we are trying to take/drop, and that by taking this lock we are not violating our defined locking order. The RESOLVE macro (used throughout the code to check whether we need to resolve an indirect reference) calls fz_assert_lock_not_held to ensure that we aren't about to resolve an indirect reference (and hence move the stream pointer) when the file is locked. In order to implement the file locking properly, pdf_open_stream (and friends) now lock the file as a side effect (because they fz_seek to the start of the stream). The lock is automatically dropped on an fz_close of such streams. Previously, the glyph cache was created in a context when it was first required; this presents problems as it can be shared between several contexts or not, depending on whether it is created before the contexts are cloned. We now always create it at startup, so it is always shared. This means that we need reference counting for the glyph caches. Added here. In fz_render_glyph, we take the glyph cache lock, and check to see whether the glyph is in the cache. If it is, we bump the refcount, drop the lock and returned the cached character. If it is not, we need to render the character. For freetype based fonts we keep the lock throughout the rendering process, thus ensuring that freetype is only called in a single threaded manner. For type3 fonts, however, we need to invoke the interpreter again to render the glyph streams. This can require reentrance to this routine. We therefore drop the glyph cache lock, call the interpreter to render us our pixmap, and take the lock again. This dropping and retaking of the lock introduces a possible race condition; 2 threads may try to render the same character at the same time. We therefore modify our hash table insert routines to behave differently if it comes to insert an entry only to find that an entry with the same key is already there. We spot this case; if we have just rendered a type3 glyph and when we try to insert it into the cache discover that someone has beaten us to it, we just discard our entry and use the cached one. Hopefully this will seldom be a problem in practise; to solve it properly would require greater complexity (probably involving spotting that another thread is already working on the desired rendering, and sleeping on a semaphore until it completes).
2012-02-06	Pass context to cmap and font descriptor functions.	Tor Andersson

2012-01-19	Multi-threading support for MuPDF	Robin Watts
	When we moved over to a context based system, we laid the foundation for a thread-safe mupdf. This commit should complete that process. Firstly, fz_clone_context is properly implemented so that it makes a new context, but shares certain sections (currently just the allocator, and the store). Secondly, we add locking (to parts of the code that have previously just had placeholder LOCK/UNLOCK comments). Functions to lock and unlock a mutex are added to the allocator structure; omit these (as is the case today) and no multithreading is (safely) possible. The context will refuse to clone if these are not provided. Finally we flesh out the LOCK/UNLOCK comments to be real calls of the functions - unfortunately this requires us to plumb fz_context into the fz_keep_storable function (and all the fz_keep_xxx functions that call it). This is the largest section of the patch. No changes expected to any test files.
2012-01-11	Use enum for FZ_STORE_DEFAULT default size.	Tor Andersson

2012-01-11	Hide glyph cache in context.	Tor Andersson

2011-12-15	Add scavenging functionality.	Robin Watts
	When fz_malloc (etc) are about to fail, we try to scavenge memory from the store and then retry. We repeatedly try to bin objects from the store until the malloc succeeds, or until we have nothing else to bin. This means we no longer need the 'aging' of the store, so this is removed.
2011-12-15	Rework pdf_store to fz_store, a part of fz_context.	Robin Watts
	Firstly, we rename pdf_store to fz_store, reflecting the fact that there are no pdf specific dependencies on it. Next, we rework it so that all the objects that can be stored in the store start with an fz_storable structure. This consists of a reference count, and a function used to free the object when the reference count reaches zero. All the keep/drop functions are then reimplemented by calling fz_keep_sharable/fz_drop_sharable. The 'drop' functions as supplied by the callers are thus now 'free' functions, only called if the reference count drops to 0. The store changes to keep all the items in the store in the linked list (which becomes a doubly linked one). We still make use of the hashtable to index into this list quickly, but we now have the objects in an LRU ordering within the list. Every object is put into the store, with a size record; this is an estimate of how much memory would be freed by freeing that object. The store is moved into the context and given a maximum size; when new things are inserted into the store, care is taken to ensure that we do not expand beyond this size. We evict any stored items (that are not in use) starting from the least recently used. Finding an object in the store now takes a reference to it already. LOCK and UNLOCK comments are used to indicate where locks need to be taken and released to ensure thread safety.
2011-12-08	Stylistic changes when testing pointer values for NULL.	Tor Andersson
	Also: use 'cannot' instead of 'failed to' in error messages.
2011-12-08	Remove remaining fz_error_note calls in the pdf code.	Tor Andersson

2011-12-06	Move antialias levels into context.	Robin Watts
	In builds that support configurable layers of antialiasing, move the variables that control this into the context. This makes it possible to safely use different levels of antialiasing in different threads.
2011-11-28	Move Freetype globals into context.	Robin Watts
	Freetype globals are not shared between threads currently - to do that we'll need to introduce a lock.