Posts by MeanMrMustard

223

Comprehensive NWT Comparison Project (calling all technically skilled members)

by Apognophos in

is anyone aware if this project is already being done by someone?

if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.

a couple initial impressions:.

MeanMrMustard

And, yeah, I too have the NWT (I ripped them from their website and processed to text only format) and tried to compare the old one to the new one. I even found a bug in the 2013 version :D Actually it will be interesting to see if they fix it when I report it here. Then again there would be some fun in reporting it directly to them too.

You want the code that I used to do the match-patch? You can get the google-diff-match-patch online, but you have to modify it to do a word-level diff. The google-diff-match-patch is a character level diff, and it works quite well. To do it for words, you have to tokenize each verse into words, then map them to a unicode character, diff the unicode char sentence, and then map back to the words. First you have to define what a "word" is... and its tougher than it looks. For me, each space is a word, and so is punctuation, as well as what you would consider a normal word.

Specifically, in Exodus 16:8, the comma after the word "satisfaction" is bolded. It is the only bolded piece of text in the entire document. I doubt it is intentional.

I am going to check this out. I didn't even pay attention to the html formatting when I scraped it from their site. I pulled all formatting out and boiled it down to text even before it hit my database. (note: just checked this out, and yes, that comma is definitely bold all by its lonesome...)

MMM

223

Comprehensive NWT Comparison Project (calling all technically skilled members)

by Apognophos in

is anyone aware if this project is already being done by someone?

if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.

a couple initial impressions:.

MeanMrMustard

Hi,

Yeah, I got distracted for a while, actually into writing some fancy new tools for reverse engineering, something I've intended to do for quite a while now but never got inspired enough by any one problem :) I guess I'm trying to figure out the format mostly out of curiosity; I agree it's likely to be much easier to get to the texts just by scripting.

I still have at least one of the lookup tables to figure out; it seems that it (probably something like a Huffman tree) is read from the .PUB files, but compressed in yet another way, with something called "nibble codec". It might start to make sense once I figure out the offset it's read from.

Anyway, some pieces about the .PUB format, just since someone is going to be curious anyway. I've been looking at a file named km1979_e.pub from the 2006 version. It contains an article named "What has happened to love?". The sha1sum of the file is 6b4c4b0c7f93c04d8aa685fb773f6826af681d35.

This is going to contain stuff that won't make much sense without referencing the contents of the file; I guess it's of value mostly if someone else wants to try to figure it out now or later.

The first 16 bytes are something called the URE header and seem to be pretty fixed:

00000000 55 52 45 53 04 00 03 00 00 00 00 00 00 00 00 00 |URES............|

I seem to remember that the 04 here tells this is a PUB file, as opposed to some other of the files (like indices, NWT, etc.), but my notes are hazy about this. 03 might be the version, which the code requires to be 03.

After that comes the PUB header, 8 bytes:

00000010 00 00 84 00 00 00 03 09

Here, I think the dword at 0x12, that is 0x84, is the number of entries in some structure which I for now call (rightly or wrongly) the Document Area Position Info Chunk. I think the 0x09 is probably the number of entries in the index data structure that immediately follows at offset 0x18 (the Chunk Information table).

At offset 0x18, the Chunk Information table. It consists of 0x09 (from previous header) 9-byte entries, each of which contains a tag of one byte and two dwords. The latter of the two dwords seems to (usually?) be an index into the document, and I guess it might make sense for the other to be the size of the element pointed to.

00000018: 00 0e 02 00 00 e6 55 00 00 02 25 00 00 00 59 00 00 00...

That is, tag 0, dwords 0x20e and 0x55e6. One of the other entries is {4, 0x261cc, 0x5ec4}. At 0x5ec4 + 0x10 (for the URE header) seems to lie some kind of structure which seems to list offsets of the chunks of the article in question; in any case, it has further pointers into the file. Inside that structure, at offset 0x6029, for example, we have the value 0x193a6, which I think points to the header of the chunk structure at something like 0x10 (for the URE header) + 0x5ec4 (for the above structure) + 0x193a6 = 0x1f27a. Anyway in 0x1f27a + 4*0x84 + 5 (I don't know yet...) = 0x1f48f some kind of header for the text chunk, with some BTEC1 (whatever) encoded information. After that comes what looks like a possible file name for the original text, 0902_K79.LOV. After that at 0x1f4a2 is what I think is a 5-byte header describing the codec used for the text, details yet unknown.

Immediately after that starts a MTEC3 (something Huffman-like) decompressed text stream. I'm not quite yet ready to say much about the compression, although I have code to do the decoding given the lookup tables (just I don't understand it yet ;). This stream seems to only contain the title, in MEPS encoding, of course.

At 0x1f760 (bytes 1b c0 ad 42) starts a MTEC3 encoded text block for which I have the lookup tables and which I thus can decompress with my code. It's a piece of text from "What has happened to love?":

". However, use your good judgment for we want the householders to know we are there because we love them and want to help them—not just to place literature. If no one is at home, leave the tract out of sight. (It is illegal to put items in the mailbox.—See Our Kingdom Service, April 1976, Announcements, page 2.) A territory can be reported as worked when we cover it with tracts." (and so on, until the end of the article.)

MEPS-encoded, the beginning of this is

. (43 08) SPACE (61 fb) H (07 08) o (28 08) w (30 08) e (1e 08) v (2f 08) e (1e 08) r (2b 08) , (44 08) SPACE (61 fb) u (2e 08) s (2c 08) e (1e 08) SPACE (61 fb) y (32 08) ...

I have to hand it to you - you have a sense of persistence. It would be cool to know the format of the PUB files. Ultimately I think they will do away with the WTLIB CD and publish the whole darn thing online from now on. As mentioned in another thread, this will allow them to electronically publish their new light without leaving any trace of what was there before - except for those sites that may log internet history, or if curios developers want to database it for fun. :)

I guess I'll post some code once I have figured out how to read/compute the LUTs :) They certainly have found a complicated way to store text...

Go for it! When I get ready, I am going to dump all of my code and stats here on this site for others to look at and learn from.

MMM

170

Sign this petition - Investigation On Jehovahs Witnesses Religious Policy That Violates Human Rights and Abuses Religious Freedom

by TJ Curioso in

launch an investigation on jehovahs witnesses religious policy that violates human rights and abuses religious freedom.

link.

MeanMrMustard

@PoconosKnows:

" My comments and points were torn apart since my original post. I am defending them - 1 by 1 -like anyone else did - when they see unfair remarks on a topic."

But that is expected in a forum like this. And yes, you are defending them 1-by-1, and that is fine. You aren't being silenced, after all you ARE defending yourself on each topic. But we have to be realistic. In other words, the petition's purpose is to ultimate stop the WT shunning doctrine. OK, nobody here agrees with that doctrine - not even from a Biblical perpective. However, getting the authorities involved is a horrible idea for all the reasons listed - you give the goverment a hammer to bash someone, then it still has the hammer, and the next thing bashed might be YOU!

MMM

170

Sign this petition - Investigation On Jehovahs Witnesses Religious Policy That Violates Human Rights and Abuses Religious Freedom

by TJ Curioso in

launch an investigation on jehovahs witnesses religious policy that violates human rights and abuses religious freedom.

link.

MeanMrMustard

I believe what is being said here is, a religion should not have the power to impose "threats" upon placing any restrictions on human rights, or it should be looked at as a form of extremism, if it results in deaths by suicide.

Define in this context "human rights". What "right" are you referring to? Is it the "right" that your family talk to you? Define this right.

What other "forms of extremism" would be included in your definition? Who decides what is "extreme"? You? Me?

When someone commits suicide, it is tragic. However, in what way should others be held legally responsible for someone killing himself? If I divorce my wife, and she kills herself after, am I now responsible for her death? Are you saying that I would need to LEGALLY be forced to stay married to prevent the possibility of her killing herself?

The JW's are a corporation - that act as a religious institution, and any corporation that imposed "hate speech" upon blacks or gays would not be tolerated - so why call people "mentally diseased" (in the present day - with the present zero tolerance of stigmatization "craze" that is sweeping our nation) those who wish to convert?

First, I believe the WTB&TS as a corporation has legally disconnected itself with the body that produces doctrine, not that it matters much.

Second, define "hate speech".

Third, hate speech against gays and blacks, or any other group, is most definitely tolerated - and should be! If it were not tolerated, free speech would be meaningless. Are you saying that members of the KKK should be fined and perhaps sent to jail because they might say horrible things about blacks and jews? Of course it is distasteful, but agreement is NOT toleration. You tolerate something you don't agree with by definition. If you agreed with it, you wouldn't have to tolerate it, because you would agree.

Tarnishing the reputation of those from their own family associations? It needs to be investigated.

Slander is a different story. But there are specific laws that define such an offense. It is not simply "that-guy-said-something-bad-about-me" sort of thing.

But I do believe, if anything is done by the legal departments of the agencies, this has to spill out to the other religions that shun, as well.

If anything is done by "legal departments" or "agencies", then it would spill out to more than religions. It would spill out to any group not currently in political power - which might include YOU!

MMM

170

Sign this petition - Investigation On Jehovahs Witnesses Religious Policy That Violates Human Rights and Abuses Religious Freedom

by TJ Curioso in

launch an investigation on jehovahs witnesses religious policy that violates human rights and abuses religious freedom.

link.

MeanMrMustard

In most modernized countries like the US for example there is a statue of freedom of religion that involves human rights.

What does that even mean? Can you define what "human right" is violated by the WTS' policy? Can you point to and quote this "statute"?

Freedom of religion shouldn't mean a religion has the right to impose hatered upon an individual simply for choosing another

religious faith, breaking up families in the process and causing assorted amount of grief and anguish.

This is what I worry about. In this case disagreement is conflated with hatred. This is such a slippery slope. I have a feeling that if you were to try to define what "hatred" is, you would find your definition would be so general that it would include simple disagreement.

MMM

34

Breyers is not allowed to call it's product "ice cream" anymore by the FDA!

by EndofMysteries in

many other so called ice cream at grocery stores as well.

have you noticed how it seems to be different than from a couple years ago?

that it doesn't melt, it just warms up and becomes like a whipped cream texture?

MeanMrMustard

Häagen-Dazs .. Strawberry.. Get some. Best ice cream EVER!

Incredients: cream, skim milk, strawberries, sugar, egg yolks

MMM

223

Comprehensive NWT Comparison Project (calling all technically skilled members)

by Apognophos in

is anyone aware if this project is already being done by someone?

if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.

a couple initial impressions:.

MeanMrMustard

@DS211:

Sent in PM.

MMM

223

Comprehensive NWT Comparison Project (calling all technically skilled members)

by Apognophos in

is anyone aware if this project is already being done by someone?

if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.

a couple initial impressions:.

MeanMrMustard

@slii, Hi there! This looks like your one and only post. I've let this thread slide, and I missed your post. Sorry about that.

I don't think the .PUB files in the Watchtower Library are in this same format. I've done lately some reverse engineering of the 2006 version, and I now understand a bit about the format of the publication files.

Good for you! I don't have the stomach to get into those PUB files. I didn't really think the PUB files were encoded the way the mobile app is encoding its data.

First of all, I have so far seen nothing to indicate that any portion of the files are encrypted. Large parts of them are compressed, though, using a compression algorithm resembling Huffman which I'm starting to understand (I still don't fully understand the construction of some lookup tables used for decompression).

You may be correct here.

Some pieces of the textual data (mainly titles) are in uncompressed form, yet this is not immediately obvious from inspecting the files. This is because the system uses internally a 16-bit MEPS-specific character set, which I believe is able to represent multiple scripts but predates Unicode. Overall, I get the impression that whoever designed this knew quite well what they were doing, but there's obviously lots of arcane legacy baggage involved. As to why WTL, or at least the 2006 version, still uses MEPS-coded documents internally, I do not know; perhaps they consider it a useful obfuscation to throw at possible reverse engineers (it did make me scratch my hea for a while), or maybe it's just a legacy thing that hasn't been enough of a problem to touch in code that could be in maintenance-only mode.

Agreed. At this point I think MEPS is dead. Unicode can take its place quite easily and, in fact, be a lot easier to work with. But I think there are a lot of things about the WT Lib that are archane (more on that below).

For example, if you look at wte.lib, it contains lots of uncompressed strings (publication names, if I remember correctly). Most (all?) .pub files also contain some uncompressed strings. In English-language files, look for places where every other byte is a 08 (hex); most of those will likely be strings. At least uncompressed MEPS strings tend to be stored in a Pascal string like style, i.e. the first 16 bits are the length of the string (in bytes, so it's always an even number), and not null-terminated.

Interesting.

Some 16-bit values seem to be mapped to some kind of control codes that most probably specify things like italics. Most of the Latin alphabet seems to be in the 08xx range. Being a typesetting-oriented coding system, it also seems to contain codes for ligatures; for example, the "ff" ligature is apparently represented as 0851.

Specifically, the English alphabet is mapped to 16-bit values, which are stored in little-endian format, as follows (numbers in hex):

0800..0819 A-Z; 081a..0833 a-z; 0834..083d numbers, "1234567890" (BTW this is the only character set I know of where 0 comes after 9, not before 1)

0841..0844 ":;.,"; 0845, 0846 left and right single quotation marks; 0847..0848 "?!"; 084b..084e "()/-"; 084f em-dash; 0850 en-dash; fb61 <SPACE>

0851 ff ligature; 0865 é (small e with acute); 0x08fb hyphenation point (used a lot e.g. in NWT to show proper hyphenation of names).

fb57 and fb58 are often seen around dashes, as in <fb57>--<fb58>. Perhaps they prevent breaking the line between them?

I guess the overall format of the .pub files and the compression are a topic for another post.

Very interesting! You have definitely gotten into the weeds. Ultimately however, I think there might be an easier way. To date I have successfully extracted every piece of text from the 2011 and 2012 WTLibs. I did this without looking at the PUB files and without any manual clicking. I planned on mentioning it in a future thread with some statistics calculated from the text.

I thought it would be cool to see if there are any meaningful differences between the 2011 and 2012 version of text. That is, we expect some differences - new content for the 2012 version, and probably some new entries into the publication index. But aside from that, I am just wondering if there are any textual changes they snuck in. The NWT project turned out to be difficult because there were so many changes involved, and it was meant to be that way. But if you take, say, the Jan 1 1980 version of the WT in the 2011 version and the 2012 version, you would expect that there are no changes between these two documents.

Anyhow, the code base is the same between the 2011 and 2012 version, and probably all the previous versions. They just add new content. I can tell because in the 2011 version has a serious memory leak in it, and the same issue is carried over to the 2012 version.

MMM

11

HALF AND HALF or 50/50

by Terry in

jw
friends

there are only a thousand people left on earth.. about ten percent of the survivors (of some unnamed disaster) have professional expertise of some kind.. 80 per cent have some education, but not any practical grasp of how things operate.. 10 per cent are extremists of varying kinds; a sort of lunatic fringe with knee-jerk "solutions" they propose to carry out.. a temporary leader/spokesman acts as a referee to sort things out.. a meeting is conducted with all 1000 present.. after much arguing, a vote is conducted.. 51 % want to carry out one set of plans while 49% want know part of it.. the thin majority start doing things their way and the minority get more and more outraged.. what happens next?.

.

MeanMrMustard

@Terry:

"The reason I ask is because it is a microcosm of what is happening in America."

I see where you might be going. Let me ask something. When you say: " A temporary leader/spokesman acts as a referee to sort things out", what exactly does "referee" mean in your scenero?

My gut reaction - if there is a rule of law (no killing, stealing, etc and an inforcement mechanism), then what happens next? Natually, captialism. The real kind, not the messed up kind we have today. There will natually be some that invest time to produce capital goods, and then become more efficient, produce more, etc.

MMM

30

Has the Jehovah's Witness computer data base number totaling 23,720 child molesters on file in Patterson, New York created a public nuisance by keeping the names secret ?

by Sol Reform in

has the jehovah's witness computer data base number totaling 23,720 child molesters on file in patterson, new york created a public nuisance by keeping the names secret ?.

that question, is basically before the court,regarding roman catholics.. .

in the first lawsuit under a new minnesota law that allows victims of child sexual abuse to bring claims decades later, accuses the archdiocese of saint paul and minneapolis and the diocese of winona of negligence for allowing a known pedophile to continue working with children for years.. the plaintiff, identified as "doe 1," seeks a court order to force the church to release an internal list of "credibly accused child molesting priests," and accuses the church of creating a public nuisance by keeping the names secret.. .

MeanMrMustard

@Simon:

Almost all authoritarian regimes keep records because information is POWER that they may need to use against someone one day ... they just can't resist hanging on to things.

If they were interested in doing the right things they would simply hand the information over to the proper authorities but instead they like to keep control of everything they can.

Ok, I can get behind this. But I have a hard time believing, if there is a DB, that it will be accessible from anywhere except the legal department at headquarters. It just seems like a major screw up for them to make a DB and yet put it on a network accessible from outside that building. This stuff about Anonymous getting the DB from hacking seems cheesy. Now, if they want to infiltrate - get into the building somehow.. or get a lawyer or IT guy working there to get the info and leak it out, that's different.

MMM