32bit/64bit programming -- an interesting problem #2

...continued

I was recently looking at the source of an open-source library. The library is supported on all popular platforms in both 32bit and 64bit. When providing a library for 32bit and 64bit platforms a new problem kicks in. ie., to make sure that the applications using this library uses the correct version of the library. ie., a 32bit application should use the 32bit version of the library and a 64bit application should use the 64bit version of the library. Obviously, it is not possible to cross link the binaries of 32bit and 64bit, and so the linker would fail if the application tried to do so. But, the difficult problem here is to restrict the application from using the wrong header files of the library. ie., a 64 bit application can inadvertantly include the 32bit headers of the library and link against the 64bit version of the library -- and it is quite possible that this will succeed even without a warning (although there are cases where this would not work).

Consider this function:

//
void __cdecl messup(struct my_struct *);
//

A 64bit translation unit that calls this function after #including a 32bit header for this function would just link fine with a 64bit library for the same function. The 32bit version of my_struct and 64bit version of my_struct shall possibly be defined differently by the library due to the data-alignment requirement between 32bit and 64bit for performance reasons (padded with extra bytes?). Thus the application assumes a different structure while the library expects a different structure. This might lead to crash. Aah!

Now that's bad. So what does it finally mean? It does mean that appropriate headers are equally important as the appropriate binaries, but unfortunately lacking the support to enforce from the building tools. To take this problem one step further, given the various data models within 64bit platforms, it is not just the platform that matters, but it is the data model.

To redefine the problem again in its final form: An application that is being built on a X data model should include the headers and libraries that were built for the X data model.

There could potentially be many ways to solve this problem. A quick answer would be to have a common header file for all data models but have ifdef'ed code for each data model in the same file. This has few drawbacks (in my opinion): declarations for all data models need to be in the same file (clutter? maintenance?); it might be very difficult (possible?) to determine the data model in the pre-processor phase, so the right set of declarations go in for compilation (afaik, there does not seem to be a pre-processor directive for the data models and depending on the pre-processor directives for the platform might be too many to handle; what about unknown platforms?).

I was actually impressed by another option that this library I talked about, had used. Actually among the 32bit and 64bit platforms, the predominant data models (LP64, LLP64, ILP32) only differ in the size of long and pointer. This library while generating its own headers (during its build time) puts in the size of the long and pointer, into the header file as it was inferred during the library's compilation. This provides an easier and reliable way of identification of the data model later for which the header was built for.

The header file generation code would be something as simple as this:

//
fprintf(header_file, "#define MYLIB_SIZEOF_LONG %d", (int) sizeof(long));
fprintf(header_file, "#define MYLIB_SIZEOF_PTR %d", (int) sizeof(void*));
//

Now that we have a means to carry forward the metadata of the data model of the library onto the headers, how do we prevent the compilation in an inappropriate data model. The idea used was simple, and should be self-explanatory. The library also added the following code to their header file:

//
static char _somearray_[sizeof(long) == MYLIB_SIZEOF_LONG ? 1 : -1];
static char _somearray2_[sizeof(void*) == MYLIB_SIZEOF_PTR ? 1 : -1];
//

If it isn't obvious, these lines declare an array of size -1 (which is illegal for compilation) incase if the sizes of long and pointer of the application didn't match with the one in the headers. Cool! that's what we need.

I feel that there are 2 tradeoffs I see with this approach:

1. Though the misuse is prevented, the error message isn't friendly. When you use a wrong header file, you get a message saying 'invalid array size' or 'invalid array subscript' or 'an array should have at least one element' etc., One might have to refer to Google to figure out the issue.

2. Two more names are added to the namespace (and 2 bytes) to the current translation unit. Usage of underscores and uncommon names might almost avoid a possibility of a name collision, but still :) I would think of a single struct having one member for each enforcement rule, so there is only 1 symbol added to the global namespace.

Any other solution??

32bit/64bit programming -- an interesting problem

After being bored of my electronics posts myself, just wanted to write something back in computer science.

Now that 64bit computers have become much common and 64bit programming is becoming a necessity, it has become a need to qualify the word programming with either 32bit or 64bit -- basically because they aren't just totally the same. There have been yesteryear days where we had to qualify 16bit vs 32bit. When I interviewed people in those times, I use to ask them the 'sizeof an integer?' and give them credit if someone asks me back if I was asking about a 16bit compiler or a 32bit compiler (at least if they ask me if it was Turbo C++ or VC++ :)) and a negative mark if the answer was 2 bytes. Slowly the trend changed, 32bit programming started dominating (ie., people had no need/exposure towards 16bit programming at all) and everyone started answering 4 bytes always and I stopped asking that question. Now it's time for the question again :) (btw, I don't claim that the 2byte to 4byte is the only difference between 16bit and 32bit; this was suppose to be a basic question to start with).

64bit programming is complicated in its own ways, primarily because of the inconsistencies in the data models. With a number of data models existing for 64bit (thank God at least only 2 are predominant), it makes it even more complicated. While Linux, Solaris, Mac (and more) are all lined up for a common data model (LP64), Microsoft is as usual onto it's own unique data model (LLP64). Although it is only Microsoft, given the dominance of Microsoft in the OS market, that is good enough to be a compatibility requirement. It is my personal opinion that Microsoft has a point here -- LLP64 invites less changes on 32bit code to become 64bit compatible. And I'm pretty sure this compatibility is going to help MS more than anybody else. Understanding the appropriate data models (and the one that is being used) is important if you are programming on a 64bit platform and it becomes even more important if you want to write code that's compatible with both 32bit and 64bit platforms.

Recently I came across an interesting problem to be thought of, specially if you are writing a library that should be source-wise compatible on both 32bit and 64bit platforms. The problem, discussion and the solution being pretty long, I would talk about it in my next post....stay tuned.

64bit Windows file redirection

64bit Windows needs to be backward compatible with 32bit Windows world, so the existing 32bit Windows apps can run fine on a 64bit Windows.This is a fair and obvious requirement.

Obviously the system DLLs have to be different for the 32bit and the 64bit apps. As of 32bit windows, the system DLLs were loaded from %windir%\system32 folder (that's why named system32?). But contrary to one's expectation of a %windir%\system64 folder, MS has chosen to use the existing system32 (mind it, it is 32) folder to contain the new 64bit system DLLs (this I believe would have seemed contrary to MS too, but they probably had a architectural constraint that forced them to do so). Ok, so what happens to the 32bit DLLs? this is even more interesting. Unfortunately MS has not renamed the 64bit versions of the existing system DLLs; so the 32bit DLLs cannot be put into the same folder as the 64bit ones (system32); yes, so there is a 64bit version of user32.dll which resides in the system32 folder (you should make sure to remember that system32 is not a 32bit folder and user32.dll does not mean a 32bit DLL -- man this is really ugly). Ok, now that we've no choice, let's get on to the business. The 32bit DLLs reside at %windir%\sysWoW64 folder (did you notice the 64?). WoW64 probably means something like 'Windows on Windows 64', so it kinda make sense once you understand what is wow64 -- but at a first glance and to many, this nomenclature is confusing. You just need to get used to it.

The problem is not over yet. A 32bit application might try to load a system DLL from %windir%\system32 folder but the 32bit DLLs actually reside in sysWoW64 folder -- so yes, MS has implemented a redirector in WOW64 that redirects any file operation on %windir%\system32 by a 32bit application to %windir%\sysWoW64. A file operation by a 64bit application isn't redirected. But what if a 32bit app wants to access the "real" system32 folder -- then it needs access %windir%\sysnative which is a special virtual directory (shall be used only by 32bit apps) which would get mapped to the "real" 64bit system32 folder. To add to the confusion, there are few folders like catroot, drivers\etc, spool etc., which are not redirected to sysWoW64 and are shared by both the 32bit and 64bit applications. A relief here is that, these files can be accessed by both %windir%\system32\x and %windir%\sysnative\x -- this at least allows the applications to use %windir%\sysnative unambiguously.

sigh!!!

Running 64bit guest on a 32bit host OS

=== This isn't something new, but worth a post. ===

Virtualization isn't something new but the concept of mixing the 32bit and 64bit software in the same machine at the same time is a milestone (atleast to me). This wouldn't be possible without a hardware support -- because a 32bit Virtual Machine Monitor (VMM) running on a 32bit host operating system has no way to make the 64bit code of the host OS to run "natively" (the processor does not understand 64bit code in 32bit mode even if it is 64bit capable).

Intel's Virtualization Technology (VT) provides support for virtualization at the hardware level. Obviously this is going to make the VMM much less complicated than earlier and provide much cleaner separation between the VMs. The VT technology offers lot of flexibility to the VMMs from the hardware level to implement virtualization. One of the most important (rather, otherwise impossible) feature is the support for mixed (32bit, 64bit) VMs to run at the sametime. Needless to say that this requires a "64bit" processor with VT technology enabled. Just VT cannot do magic on a 32bit CPU. In simple terms, VT allows the VMM to switch the processor mode to 32bit or 64bit at run-time as required so the guest OS does not have the restriction on the mode of the host OS.

My most favourite VM solution is VMware Workstation; while my favourite free one being VirtualBox. Both the solutions support VT and can run 64bit guest on 32bit host. (Vmware player is actually free -- but does not allow you to create VMs but only run; Infact VMware client to their ESX server is now open-source -- more info).

VT feature has to be enabled at the BIOS -- might be available in some advanced configuration. I was surprised to see this option on my machine under 'Security' section in BIOS :)

Anyways, this is just wonderful!!!