is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
Apognophos,
I have a dumped DB for you. It is 14.1 MB zipped. I exported each verse into one file, one for the 1984 version, and one for the new 2013 version. The files are grouped under directories for each book and chapter. For example, this is what Genesis 1:5 looks like:
What is the best way to get this to you?
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
All,
I just had another thought about why the WT might want to do this - to hide the fact that MEPS is dead. MEPS was the WTB&TS' publishing system. It allowed them to print in multiple languages and distribute the literature world-wide at a time when that sort of thing was difficult. But when MEPS (Multilanguage Electronic Phototypesetting System) became MEPS (Multilanguage Electronic Publishing System), the flagship publishing system was reduced to a simple Unicode text file. But they can make it look like something else, something special. They can make it look like it's not just a text file. It's a MEPS blob of binary data from an advanced printing system - *wink* *wink*... *nudge* *nudge*
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
Apognophos,
Absolutely! My next task, before I do any diff-match-patch stuff is to dump each verse out to its own text file - one for the old version and one for the new. I can probably do that tonight sometime :)
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
google-diff-match-patch is great and there's lots of support for it. Not sure if word level is necessary - doing a line level diff (i.e. each verse) is probably better for a first pass and then more specific diffs can be done of those later if needed but most times someone will need to read the whole verse and see the change, less so single words in isolation.
Simon,
Hmmm, you may be right. I may be confusing what is meant by "line" level and "word" level. I know the google-diff-match-patch is, by default, a character level diff. I am interpreting a "line level" to mean that the diff will act upon two entire lines - essencially indicating which lines are different. Since, as you note, all these are on one single line (each verse), they are always going to be different on that level. I am interpreting the "word" level to be differences in the groupings of characters separated by whitespace of some sort (words). Like I said, I could have it wrong and what we need is a line level. The good thing is that google-diff-match-match seems to be - as you state - a lot of support. And moving between the two shouldn't be hard. We could do both and see how it works out.
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
All,
More to come later... have to get back to my class...
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
In my limited knowledge, it seems to me that MMM and Simon are saying that they protected it stupid. They encrypted each verse separately, but with the same code. Once the code is broken, the entire package unwraps. But all that extra coding and encrypting means the PDF bible is far bigger than it needs to be.
Now, I am not in IT, so if I got anything of this wrong, feel free to correct me.
Well... the only reason I was able to get this was because android app run from a form of java, and you can decompile that. That means I can take the program and reverse the code out of it. Normally, java (.class files) are much easier to decompile. The android java gets mangled down so that some of the code is permanently lost. However, it can be decompiled to an assembly-like code. The logic is still there, but its not very easy to read. It took me a couple nights to get through it. Once I saw what was going on, it was all downhill from there.
Each publication will have a different SHA-256 hash because that hash value is derived from the language + pub symbol + year. So even different languages would be encrypted a little differently - but the overall algorithm is the same.
The PDF is public. There was nothing encrypted there. I was reading the PDF with some standard PDF libraries. The issue with the PDF was that it was horrible to parse through given the way PDFs are built.
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
For the non-cognoscente, what are the practical uses of this hack?
How does the Ex-JW community benefit?
If it proves to be very useful, shouldn't somebody use the temporary vulnerability to maximum effect before it gets locked down and changed?
I don't know about anyone else, but it gave me this sort of feeling toward the WT:
http://www.youtube.com/watch?v=FWBUl7oT9sA
Benefical? Emotionally - very much so.
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
MMM - did it seem to you the WTS had gone to extra lengths to prevent what you have done? Speculate why?
It is somewhat reasonable to think that they don't want to let other developers create programs against their data. So if they come out with an android app for the NWTx2, the WT wants people to download their app, not Mr. ex-JW that might link to other resources from jwfacts.com. :)
But then I catch myself... if that was their reason, they undermine the entire thing by posting the Bible online. In the end, even after getting past the DB encryption, I ended up just taking the text from the web version. Why? It was a lot easier - the text is right there for anyone to get. All I had to do was download it via HTTP and use standard objects to pull the text from the HTML. Why does this matter? Well, if I were making an android (or PC) app, I don't have to hack anything - I would simply link to the scripture and download it live from jw.org (and then insert commentary from jwfacts).
Ultimately, I don't know what it gains them to encrypt it.
As I mentioned before, even the WT library is vunerable. I don't need to know how the PUB files in the WB library are formatted to read the content there. Why? Because I know that the WT library program itself will decrypt it and display it in a window, and I know that you can copy the text to the clipboard. You think that can't be automted? PFFT! It surely can. I can create my own WT lib database if I wanted to without knowing any of the encryption.
So you might be asking why I spent time cracking the DB in the first place? Because the NWTx2 is new and they didn't have an easier way of getting it. The PDF was horrible to parse through, and it looked like my only choice until castthefirststone brought our attention to the android DB of it. If the DB was cracked then I could pull the text out of there without all of the PDF parsing issues that would most likely cause errors to appear in the text.
But, at the last moment, the WT put out an online version...
MMM
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
As another update! - DONE! I have both versions. Now on to diffing th text.
Apognophos, I was thinking of using google-diff-match-patch. There are articles on turning it into word level diff with just a short patch. Plus it seems to be more of what we want: http://code.google.com/p/google-diff-match-patch/
I am also going to try to work on dumping my DB to files, one file, one verse. There will be 62,206 files when all is said and done. I'm not sure how to share such a thing at the moment...
MMM
(also: my last two posts, made late, had a few spelling mistakes. When I said that John 8 1-11 are "not left completely out" of 2013 version, the "not" should be "now". " partse" should be "parse" )
is anyone aware if this project is already being done by someone?
if not, perhaps we can brainstorm how to go about this, now that the pdf for the revised nwt is out.
a couple initial impressions:.
As an update:
I am gathering the text to the NWTx2 version and stumbled upon some issues with John 7 and 8. The portions of chapter 8 (verses 1-11) that are printed in the 1984 version with a warning that they aren't found in some manuscripts are not completely left out in the 2013 version. I didn't expect this and its giving my parser some issues... fixing now.
MMM
